# Frontier comparison — Haiku 4.5 & Sonnet 4.6 (sub-agent runs)

Generated 2026-06-20. Same 140-item battery, same `vibe_eval.grade()`, closed-book,
no tools, reasoning only. Claude models run as Claude Code sub-agents (no API key/credits),
so only GENERATION differs from VibeThinker's HTTP runs — grading is identical.

## Final three-way grid (avg@1, %)

| Model              | AIME24 | AIME25 | HMMT25 | GPQA |
|--------------------|:------:|:------:|:------:|:----:|
| VibeThinker-3B     |  93.3  |  93.3  |  70.0  | 62.0 |
| Claude Haiku 4.5   |  33.3  |  20.0  |  10.0  | 64.0 |
| Claude Sonnet 4.6  |  96.7  |  83.3  |  83.3  | 80.0 |

Counts: Sonnet aime24 29/30, aime25 25/30, hmmt25 25/30, gpqa 40/50.
Haiku scored from the prior session's batches (aggregate only retained).

## Methodology / validity controls
- Golds withheld: `/tmp/haiku_b/` batch files have NO answers; key moved to `/tmp/haiku_gold.json`.
- Compute lockdown: every agent showed tool_uses ≤ 1 (just one Read) — no Bash/Python cheating.
- Grading: each prediction wrapped as `\boxed{<pred>}` and passed to `vibe_eval.grade()`
  (integer-exact for AIME, symbolic math-verify equivalence for HMMT non-integer golds, letter-match for GPQA).
- Claude ran WITHOUT extended thinking (in-context CoT only); Sonnet still scored 83–97 on math → setup is fair.
- 32k-output-token ceiling forced the hardest AIME25/HMMT25 batches to be split to ONE problem per agent.

## CAVEAT (must stay in any writeup)
Claude models are capability REFERENCE POINTS, **not contamination-controlled** — zero visibility
into their training, and AIME/GPQA are public. Treat their scores as a generous bound, not a clean one.

## The 10 recovered Sonnet items (graded this session, one-problem-per-agent)
These were the hardest tier (left over from a session limit). Sonnet went aime25 0/5, hmmt25 1/5,
which is what dropped its aime25 100→83.3 and hmmt25 96→83.3 vs the earlier partial table.

| id           | Sonnet pred        | gold              | correct |
|--------------|--------------------|-------------------|:-------:|
| aime25-010   | 16                 | 259               |   no    |
| aime25-011   | 678                | 510               |   no    |
| aime25-012   | 162                | 204               |   no    |
| aime25-013   | 59                 | 60                |   no    |
| aime25-014   | 683                | 735               |   no    |
| hmmt25-015   | 6300               | 6300              |  YES    |
| hmmt25-016   | 26!                | 2^{25} \cdot 26!  |   no    |
| hmmt25-017   | \frac{675}{67}     | \frac{2025}{101}  |   no    |
| hmmt25-018   | \frac{3}{4}        | \frac{4}{9}       |   no    |
| hmmt25-019   | \frac{945}{2}      | \frac{448}{3}     |   no    |

Note: the other ~120 Haiku/Sonnet per-item predictions live only in this session's
sub-agent transcript files under the Claude Code tasks dir (not re-exported here).