HumanEval · distilled
89%
pass@1, executed
It didn't — not even at its own job. This 12B was distilled for Python coding, from >1-trillion-parameter frontier teachers (Composer 2.5 + Claude Fable 5). Yet Google's plain gemma-4-12B beat it on executable code — 96% vs 89% pass@1 on HumanEval — and never lost a reasoning category besides. Here is exactly what we ran, and how we know.
Both are 12B parameters at Q8 — so this isolates the distillation effect, not a size advantage. The only difference that should matter is where the weights learned to think.
Every item is identical for both models, seed-fixed, with the served model logged per answer. One set is the model's home turf; the other four ask whether the distillation bought reasoning that transfers. We show all of it — on-domain and off — so the result can't be waved away as the wrong test.
Overall accuracy, distilled vs stock, under each thinking condition. The bars fill to score.
| Category | dist · off | stock · off | dist · on | stock · on |
|---|---|---|---|---|
| BBH · logic | 100% | 100% | 100% | 100% |
| GSM8K · math | 90% | 97% | 97% | 97% |
| MATH-500 | 85% | 95% | 70% | 90% |
| MMLU-Pro | 73% | 80% | 60% | 67% |
| OVERALL | 88% | 94% | 85% | 91%floor |
Reasoning is off-domain for this model — its card calls it a fine-tune on verifiable Python coding, distilled from solutions that were run against tests and kept only if they passed. So the fair test isn't trivia; it's code. We ran HumanEval — 164 problems, each candidate executed against hidden unit tests in a sandbox. pass@1, greedy. It lost here too — and this time the gap is statistically real.
A clean-looking benchmark can hide four different lies. Each was checked before the verdict was trusted.
A reasoning-distilled model loves to show its work, which can fake a lead. So both models got the identical chain-of-thought prompt and the same ANSWER: contract — a distilled win would mean reasoning, not just verbosity.
An earlier config quietly collapsed both model IDs onto a single resident file. After that, every response's served-model id was logged per item — so we can prove the distilled model wasn't benchmarked against itself. Provenance is in the results file.
With thinking on, the first run's MMLU-Pro cratered to 47% — but two answers had simply run out of tokens mid-thought and scored zero with no ANSWER:. We raised the budget to 16k and re-ran. A silent truncation would have invented a fake collapse.
The whole pitch of distilling from a reasoning teacher lives in the thinking channel — so we tested it on. Both models genuinely produced reasoning traces. The distilled model still lost, and thinking lowered its score rather than raising it.
On two free-response MATH problems, the distilled model answered correctly when replying directly — then, with thinking on, deliberated its way to a multiple-choice letter on a question that had no options at all.
Average tokens spent per MMLU-Pro answer with thinking on. Stock deliberates ~5× longer — and still only reaches a floor of 67%. Length is not the lever.
The distilled model's one genuine difference is brevity — it answers in a fraction of the tokens. That conciseness kept it close on easy problems, but it isn't a reasoning advantage: on the hard ones it still fell behind. Distillation here compressed the model, it didn't sharpen its reasoning.
On the reasoning suite (N = 85), the 6-point gaps are ~1.4σ — directional, not significant in isolation; confidence there comes from the same direction repeating across all four cells. The coding result is the firm one: N = 164, significant on its own (p ≈ 0.006).
We tested both off-domain (reasoning) and on-domain (HumanEval — what it was built for). It lost both. So this isn't an off-domain gotcha — but it's still one checkpoint, one quant; not a universal law that distillation can't help.
Stock's thinking-on score (91%) is suppressed by 6 truncated items. Fixing them only moves stock up — it never flips the result.
Add MBPP+ and HumanEval+ (extra hidden tests), plus a thinking-on coding pass. The HumanEval gap is already significant (p ≈ 0.006) — more data widens it, it doesn't flip.
Four scripts, the exact seeded items both models saw, and every per-item prediction tagged with the served_model that produced it. Read the code below, or take the bundle and re-score it yourself. The only redaction is the llama-swap host and a local path.
#!/usr/bin/env python3
"""Fetch seeded reasoning-eval samples -> data/*.jsonl. Network only (no model server)."""
import json, random, os, sys
from datasets import load_dataset
OUT = os.path.join(os.path.dirname(__file__), "data")
os.makedirs(OUT, exist_ok=True)
SEED = 1234
# (name, n) — directional defaults; override via argv: prep_data.py gsm8k=100 math=50 ...
DEFAULTS = {"gsm8k": 30, "math": 20, "bbh": 20, "gpqa": 15}
overrides = dict(kv.split("=") for kv in sys.argv[1:] if "=" in kv)
N = {k: int(overrides.get(k, v)) for k, v in DEFAULTS.items()}
def save(name, rows):
p = os.path.join(OUT, f"{name}.jsonl")
with open(p, "w") as f:
for r in rows: f.write(json.dumps(r) + "\n")
print(f" saved {len(rows):>4} -> {p}")
def sample(ds, n):
idx = list(range(len(ds)))
random.Random(SEED).shuffle(idx)
return [ds[i] for i in idx[:n]]
print("Fetching eval samples (seed=%d)..." % SEED)
# GSM8K — answer after '#### '
try:
ds = load_dataset("openai/gsm8k", "main", split="test")
rows = [{"id": f"gsm8k-{i}", "category": "gsm8k",
"question": r["question"],
"answer": r["answer"].split("####")[-1].strip().replace(",", "")}
for i, r in enumerate(sample(ds, N["gsm8k"]))]
save("gsm8k", rows)
except Exception as e:
print(f" GSM8K FAILED: {e}")
# MATH-500 — clean ungated test set with explicit 'answer' field
try:
ds = load_dataset("HuggingFaceH4/MATH-500", split="test")
rows = [{"id": f"math-{i}", "category": "math",
"question": r["problem"], "answer": str(r["answer"]).strip(),
"level": r.get("level"), "type": r.get("subject")}
for i, r in enumerate(sample(ds, N["math"]))]
save("math", rows)
except Exception as e:
print(f" MATH FAILED: {e}")
# BBH — use a multi-step subset (causal_judgement is y/n; use 'logical_deduction_three_objects' MC)
try:
ds = load_dataset("lukaemon/bbh", "logical_deduction_three_objects", split="test")
rows = [{"id": f"bbh-{i}", "category": "bbh",
"question": r["input"], "answer": r["target"].strip("()").strip()}
for i, r in enumerate(sample(ds, N["bbh"]))]
save("bbh", rows)
except Exception as e:
print(f" BBH FAILED: {e} (may need a different config name)")
# MMLU-Pro — ungated, 10-option hard knowledge+reasoning (harder than MMLU/GPQA-easy)
try:
ds = load_dataset("TIGER-Lab/MMLU-Pro", split="test")
# focus on reasoning-heavy categories
hard = ds.filter(lambda r: r["category"] in
{"physics", "engineering", "math", "chemistry", "law"})
rows = []
for i, r in enumerate(sample(hard, N["gpqa"])):
labels = "ABCDEFGHIJ"
q = r["question"] + "\n" + "\n".join(
f"{labels[k]}) {opt}" for k, opt in enumerate(r["options"]))
rows.append({"id": f"mmlupro-{i}", "category": "mmlu_pro",
"question": q, "answer": r["answer"], "subject": r["category"]})
save("mmlu_pro", rows)
except Exception as e:
print(f" MMLU-Pro SKIPPED: {e}")
print("Done. Files in", OUT)
#!/usr/bin/env python3
"""Fetch HumanEval -> data_code/humaneval.jsonl.
Each row: task_id, prompt (signature+docstring), canonical_solution (body),
test (defines check()), entry_point (function name).
Usage:
python3 prep_code.py # all 164
python3 prep_code.py 20 # first 20 (smoke test)
"""
import json, os, sys
from datasets import load_dataset
HERE = os.path.dirname(__file__)
OUT = os.path.join(HERE, "data_code"); os.makedirs(OUT, exist_ok=True)
def main():
n = int(sys.argv[1]) if len(sys.argv) > 1 else 0 # 0 = all
ds = load_dataset("openai/openai_humaneval", split="test")
rows = [{"task_id": r["task_id"], "prompt": r["prompt"],
"canonical_solution": r["canonical_solution"],
"test": r["test"], "entry_point": r["entry_point"]} for r in ds]
if n: rows = rows[:n]
p = os.path.join(OUT, "humaneval.jsonl")
with open(p, "w") as f:
for r in rows: f.write(json.dumps(r) + "\n")
print(f"wrote {len(rows)} -> {p}")
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Run reasoning eval against an OpenAI-compatible /v1/chat/completions endpoint.
Usage:
python3 run_eval.py --base http://LLAMA_SWAP_HOST:PORT --model <id> --tag distilled
python3 run_eval.py --base ... --model <id> --tag stock --key sk-...
Then compare:
python3 run_eval.py --compare results/distilled.jsonl results/stock.jsonl
"""
import json, os, re, time, argparse, urllib.request, glob
from concurrent.futures import ThreadPoolExecutor
HERE = os.path.dirname(__file__)
DATA = os.path.join(HERE, "data")
RES = os.path.join(HERE, "results"); os.makedirs(RES, exist_ok=True)
# Identical for both models — measures reasoning ABILITY, not default verbosity.
SYS = ("You are a careful problem solver. Think step by step, then end your reply "
"with a single final line in exactly this format:\nANSWER: <your final answer>\n"
"For multiple-choice, the final answer is the single letter of the correct option.")
def load_items():
items = []
for f in sorted(glob.glob(os.path.join(DATA, "*.jsonl"))):
with open(f) as fh:
items += [json.loads(l) for l in fh if l.strip()]
return items
def call(base, model, key, question, max_tokens, temp, timeout, top_p=0.95, think=False):
payload = {"model": model, "temperature": temp, "top_p": top_p,
"max_tokens": max_tokens,
"messages": [{"role": "system", "content": SYS},
{"role": "user", "content": question}]}
if think:
payload["chat_template_kwargs"] = {"enable_thinking": True}
body = json.dumps(payload).encode()
req = urllib.request.Request(base.rstrip("/") + "/v1/chat/completions", data=body,
headers={"Content-Type": "application/json",
**({"Authorization": f"Bearer {key}"} if key else {})})
t0 = time.time()
with urllib.request.urlopen(req, timeout=timeout) as r:
d = json.load(r)
dt = time.time() - t0
msg = d["choices"][0]["message"]
text = msg.get("content") or msg.get("reasoning_content") or ""
usage = d.get("usage", {})
return text, usage.get("completion_tokens"), dt, d.get("model")
def extract(text):
m = list(re.finditer(r"ANSWER:\s*(.+)", text, re.I))
if m: return m[-1].group(1).strip()
b = list(re.finditer(r"\\boxed\{([^}]*)\}", text))
if b: return b[-1].group(1).strip()
return text.strip().splitlines()[-1].strip() if text.strip() else ""
def norm_num(s):
s = s.replace(",", "").replace("$", "").replace("\\", "").strip().rstrip(".")
m = re.search(r"-?\d+\.?\d*", s)
return m.group(0) if m else None
def norm_text(s):
return re.sub(r"[\s${}\\]", "", s).lower()
def score(cat, pred, gold):
p = extract(pred) if False else pred # pred already extracted by caller
if cat in ("bbh", "mmlu_pro"):
up = p.upper()
m = re.search(r"\b([A-J])\b", up) or re.search(r"([A-J])\)", up) or re.search(r"[A-J]", up)
return bool(m) and m.group(1 if m.re.groups else 0) == gold.strip().upper()
if cat == "gsm8k":
a, b = norm_num(p), norm_num(gold)
if a is None or b is None: return False
try: return abs(float(a) - float(b)) < 1e-6
except: return a == b
# math: numeric if possible, else normalized string
a, b = norm_num(p), norm_num(gold)
if a is not None and b is not None:
try:
if abs(float(a) - float(b)) < 1e-6: return True
except: pass
return norm_text(p) == norm_text(gold)
def run(args):
items = load_items()
out = os.path.join(RES, f"{args.tag}.jsonl")
print(f"[{args.tag}] {len(items)} items -> {args.model} @ {args.base}")
def work(it):
try:
text, ctoks, dt, served = call(args.base, args.model, args.key, it["question"],
args.max_tokens, args.temp, args.timeout, args.top_p,
args.think)
pred = extract(text)
ok = bool(score(it["category"], pred, it["answer"]))
return {**{k: it[k] for k in ("id", "category", "answer")},
"pred": pred, "correct": ok, "completion_tokens": ctoks,
"latency_s": round(dt, 2), "served_model": served, "raw": text}
except Exception as e:
return {**{k: it[k] for k in ("id", "category", "answer")},
"pred": None, "correct": False, "error": str(e)}
rows = []
with ThreadPoolExecutor(max_workers=args.concurrency) as ex:
for i, r in enumerate(ex.map(work, items), 1):
rows.append(r)
flag = "ok " if r["correct"] else ("ERR" if r.get("error") else "x ")
print(f" [{i:>3}/{len(items)}] {flag} {r['id']:<14} "
f"gold={str(r['answer'])[:18]:<18} pred={str(r['pred'])[:24]}")
with open(out, "w") as f:
for r in rows: f.write(json.dumps(r) + "\n")
served = sorted({r.get("served_model") for r in rows if r.get("served_model")})
print(f"\nSERVED MODEL(S) this run: {served}")
summarize(rows, args.tag)
print(f"saved -> {out}")
def summarize(rows, tag):
cats = {}
for r in rows:
c = cats.setdefault(r["category"], {"n": 0, "ok": 0, "tok": 0, "tn": 0})
c["n"] += 1; c["ok"] += int(r["correct"])
if r.get("completion_tokens"): c["tok"] += r["completion_tokens"]; c["tn"] += 1
print(f"\n=== {tag} ===")
print(f"{'category':<12}{'acc':>10}{'avg_tok':>10}")
tot_n = tot_ok = 0
for c, v in sorted(cats.items()):
tot_n += v["n"]; tot_ok += v["ok"]
at = (v["tok"] / v["tn"]) if v["tn"] else 0
print(f"{c:<12}{v['ok']:>3}/{v['n']:<3}{100*v['ok']/v['n']:>5.0f}%{at:>10.0f}")
print(f"{'OVERALL':<12}{tot_ok:>3}/{tot_n:<3}{100*tot_ok/tot_n:>5.0f}%")
def compare(a, b):
def load(p): return {r["id"]: r for r in (json.loads(l) for l in open(p))}
A, B = load(a), load(b)
ta, tb = os.path.basename(a).split(".")[0], os.path.basename(b).split(".")[0]
cats = {}
for i in A:
if i not in B: continue
c = cats.setdefault(A[i]["category"], [0, 0, 0])
c[0] += 1; c[1] += int(A[i]["correct"]); c[2] += int(B[i]["correct"])
print(f"{'category':<12}{ta:>10}{tb:>10}{'Δ':>8}")
TA = TB = TN = 0
for c, (n, x, y) in sorted(cats.items()):
TN += n; TA += x; TB += y
print(f"{c:<12}{100*x/n:>9.0f}%{100*y/n:>9.0f}%{100*(x-y)/n:>+7.0f}")
print(f"{'OVERALL':<12}{100*TA/TN:>9.0f}%{100*TB/TN:>9.0f}%{100*(TA-TB)/TN:>+7.0f}")
print(f"\nN={TN}. Note: ~±{int(100*(0.5/ (TN**0.5))) }pt rough 1-sigma per model; "
"treat small gaps as noise.")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--base"); ap.add_argument("--model"); ap.add_argument("--key")
ap.add_argument("--tag", default="run")
ap.add_argument("--max-tokens", type=int, default=8192, dest="max_tokens")
ap.add_argument("--temp", type=float, default=0.6)
ap.add_argument("--top-p", type=float, default=0.95, dest="top_p")
ap.add_argument("--timeout", type=int, default=300)
ap.add_argument("--think", action="store_true", help="send chat_template_kwargs enable_thinking=true")
ap.add_argument("--concurrency", type=int, default=4)
ap.add_argument("--compare", nargs=2)
a = ap.parse_args()
if a.compare: compare(*a.compare)
else: run(a)
#!/usr/bin/env python3
"""Execution-graded code eval (HumanEval pass@1) against an OpenAI-compatible endpoint.
The model is asked to complete a function; we extract its code block, assemble it with
the task's hidden test, and RUN it under a layered sandbox:
- reliability_guard: neuters destructive syscalls (os.system, subprocess, rmtree, unlink, ...)
- sandbox-exec '(allow default)(deny network*)': blocks all network (macOS Seatbelt)
- fresh temp cwd, PYTHONDONTWRITEBYTECODE, hard timeout w/ SIGKILL
ALWAYS run --selftest first: it executes the CANONICAL solutions and must score ~100%.
If it doesn't, the executor is broken and no model number means anything.
Usage:
python3 run_code.py --selftest
python3 run_code.py --base http://LLAMA_SWAP_HOST:PORT --model <distilled-id> --tag code-distilled --key $API_KEY
python3 run_code.py --base http://LLAMA_SWAP_HOST:PORT --model <stock-id> --tag code-stock --key $API_KEY
python3 run_code.py --compare results_code/code-distilled.jsonl results_code/code-stock.jsonl
"""
import json, os, re, sys, time, argparse, tempfile, subprocess, urllib.request
from concurrent.futures import ThreadPoolExecutor
HERE = os.path.dirname(__file__)
DATA = os.path.join(HERE, "data_code")
RES = os.path.join(HERE, "results_code"); os.makedirs(RES, exist_ok=True)
SYS = ("You are an expert Python programmer. Complete the function described by the user. "
"You may briefly reason about edge cases, complexity, and approach, but you MUST end "
"your reply with the complete, runnable function inside a single ```python code block. "
"Include the full function definition (signature and body) and any needed imports. "
"Do not include example usage, explanation after the code, or tests.")
# Neutered inside the executed subprocess, before any candidate code runs.
GUARD = r"""
import os, shutil, subprocess, builtins, faulthandler
faulthandler.disable()
builtins.exit = None
builtins.quit = None
for _n in ("system","popen","remove","removedirs","rmdir","unlink","rename","renames",
"replace","truncate","kill","killpg","fork","forkpty","chmod","chown","chroot"):
if hasattr(os, _n): setattr(os, _n, None)
shutil.rmtree = None; shutil.move = None
subprocess.Popen = None; subprocess.run = None; subprocess.call = None
"""
def load_items():
with open(os.path.join(DATA, "humaneval.jsonl")) as f:
return [json.loads(l) for l in f if l.strip()]
# ---------- generation ----------
def call(base, model, key, prompt, max_tokens, temp, timeout, think=False):
payload = {"model": model, "temperature": temp, "top_p": 1.0, "max_tokens": max_tokens,
"messages": [{"role": "system", "content": SYS},
{"role": "user", "content": prompt}]}
if think:
payload["chat_template_kwargs"] = {"enable_thinking": True}
req = urllib.request.Request(base.rstrip("/") + "/v1/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json",
**({"Authorization": f"Bearer {key}"} if key else {})})
t0 = time.time()
with urllib.request.urlopen(req, timeout=timeout) as r:
d = json.load(r)
dt = time.time() - t0
ch = d["choices"][0]
msg = ch["message"]
text = msg.get("content") or msg.get("reasoning_content") or ""
usage = d.get("usage", {})
return text, usage.get("completion_tokens"), ch.get("finish_reason"), dt, d.get("model")
def extract_code(text):
blocks = re.findall(r"```(?:python|py)?\s*\n(.*?)```", text, re.S)
if blocks:
return blocks[-1]
m = re.search(r"(?:^|\n)(?:import |from |def |class )", text)
return text[m.start():] if m else text
# ---------- execution ----------
def build_program(item, code):
ep = item["entry_point"]
has_def = re.search(r"\bdef\s+" + re.escape(ep) + r"\b", code)
body = code if has_def else (item["prompt"] + "\n" + code)
return body + "\n\n" + item["test"] + f"\n\ncheck({ep})\n"
def execute(program, timeout, use_sandbox=True):
src = GUARD + "\n" + program
with tempfile.TemporaryDirectory() as td:
fp = os.path.join(td, "cand.py")
with open(fp, "w") as f:
f.write(src)
py = [sys.executable, "-I", "-B", fp]
cmd = (["sandbox-exec", "-p", "(version 1)(allow default)(deny network*)"] + py) if use_sandbox else py
env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1", "OMP_NUM_THREADS": "1"}
try:
r = subprocess.run(cmd, cwd=td, env=env, timeout=timeout,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
return (r.returncode == 0), ("" if r.returncode == 0 else r.stderr.decode("utf-8", "replace")[-240:])
except subprocess.TimeoutExpired:
return False, "timeout"
except Exception as e:
return False, f"exec-error: {e}"
# ---------- self-test: canonical solutions must pass ----------
def selftest(timeout, use_sandbox):
items = load_items()
print(f"[selftest] executing {len(items)} CANONICAL solutions "
f"(sandbox={'on' if use_sandbox else 'off'})")
ok = 0; fails = []
for it in items:
prog = build_program(it, it["canonical_solution"])
passed, err = execute(prog, timeout, use_sandbox)
ok += passed
if not passed: fails.append((it["task_id"], err))
print(f"[selftest] {ok}/{len(items)} canonical solutions pass")
for tid, err in fails[:10]:
print(f" FAIL {tid}: {err}")
if ok < len(items):
print("\n!! executor not clean — fix before trusting model numbers.")
if use_sandbox:
print(" (retry with --no-sandbox to check if Seatbelt is the cause)")
return ok == len(items)
# ---------- run a model ----------
def run(args):
items = load_items()
if args.limit: items = items[:args.limit]
out = os.path.join(RES, f"{args.tag}.jsonl")
print(f"[{args.tag}] {len(items)} items -> {args.model} @ {args.base}")
def work(it):
try:
text, ctoks, finish, dt, served = call(args.base, args.model, args.key, it["prompt"],
args.max_tokens, args.temp, args.timeout, args.think)
code = extract_code(text)
passed, err = execute(build_program(it, code), args.exec_timeout, not args.no_sandbox)
return {"task_id": it["task_id"], "entry_point": it["entry_point"],
"passed": bool(passed), "finish_reason": finish, "exec_err": err,
"completion_tokens": ctoks, "latency_s": round(dt, 2),
"served_model": served, "raw": text}
except Exception as e:
return {"task_id": it["task_id"], "entry_point": it["entry_point"],
"passed": False, "error": str(e)}
rows = []
with ThreadPoolExecutor(max_workers=args.concurrency) as ex:
for i, r in enumerate(ex.map(work, items), 1):
rows.append(r)
flag = "ok " if r["passed"] else ("ERR" if r.get("error") else "x ")
trunc = " [TRUNC]" if r.get("finish_reason") == "length" else ""
print(f" [{i:>3}/{len(items)}] {flag} {r['task_id']:<16} {r.get('exec_err','')[:40]}{trunc}")
with open(out, "w") as f:
for r in rows: f.write(json.dumps(r) + "\n")
served = sorted({r.get("served_model") for r in rows if r.get("served_model")})
npass = sum(r["passed"] for r in rows)
trunc = sum(1 for r in rows if r.get("finish_reason") == "length")
avgtok = ([r["completion_tokens"] for r in rows if r.get("completion_tokens")] or [0])
print(f"\nSERVED MODEL(S) this run: {served}")
print(f"=== {args.tag} ===")
print(f"pass@1: {npass}/{len(rows)} = {100*npass/len(rows):.1f}%")
print(f"avg completion tokens: {sum(avgtok)//len(avgtok)} truncated (finish=length): {trunc}")
if trunc:
print(f" !! {trunc} items hit max_tokens — likely false zeros. Re-run with higher --max-tokens.")
print(f"saved -> {out}")
def compare(a, b):
def load(p): return {r["task_id"]: r for r in (json.loads(l) for l in open(p))}
A, B = load(a), load(b)
ta, tb = os.path.basename(a).split(".")[0], os.path.basename(b).split(".")[0]
ids = [i for i in A if i in B]
pa = sum(A[i]["passed"] for i in ids); pb = sum(B[i]["passed"] for i in ids)
n = len(ids)
print(f"{'metric':<16}{ta:>16}{tb:>16}")
print(f"{'pass@1':<16}{100*pa/n:>15.1f}%{100*pb/n:>15.1f}%")
print(f"{'solved':<16}{pa:>16}{pb:>16} (N={n})")
only_a = [i for i in ids if A[i]["passed"] and not B[i]["passed"]]
only_b = [i for i in ids if B[i]["passed"] and not A[i]["passed"]]
print(f"\nsolved only by {ta}: {len(only_a)} -> {', '.join(only_a[:12])}")
print(f"solved only by {tb}: {len(only_b)} -> {', '.join(only_b[:12])}")
sig = int(100 * (0.5 / (n ** 0.5)))
print(f"\nN={n}. Rough ±{sig}pt 1-sigma per model; treat small gaps as directional.")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--base"); ap.add_argument("--model"); ap.add_argument("--key")
ap.add_argument("--tag", default="code-run")
ap.add_argument("--max-tokens", type=int, default=6144, dest="max_tokens")
ap.add_argument("--temp", type=float, default=0.0)
ap.add_argument("--timeout", type=int, default=300, help="HTTP timeout per request")
ap.add_argument("--exec-timeout", type=int, default=10, dest="exec_timeout", help="per-candidate run timeout")
ap.add_argument("--think", action="store_true")
ap.add_argument("--concurrency", type=int, default=4)
ap.add_argument("--limit", type=int, default=0)
ap.add_argument("--no-sandbox", action="store_true", help="drop sandbox-exec (reliability_guard only)")
ap.add_argument("--selftest", action="store_true")
ap.add_argument("--compare", nargs=2)
a = ap.parse_args()
if a.selftest: sys.exit(0 if selftest(a.exec_timeout, not a.no_sandbox) else 1)
elif a.compare: compare(*a.compare)
else: run(a)