The honest ledger.
Three views. Our internal synthetic suite — 7,176 items across 29 shards — the standard open benchmarks, and the attempt-by-attempt ledger: every 0.0.1 training run measured against the frozen seed, reported exactly as run. Real numbers where we have them, “pending” where we don’t — including the embarrassing ones.
67.95%
weighted · Tutte 0.0.1 beta · attempt 3 · best official bench
29.28%
weighted · 0.0.0 seed
7,176
benchmark items · 29 shards
Fig. 1 — standard open benchmarks
Open benchmarks
MMLU, HellaSwag, ARC-Challenge, GSM8K, HumanEval — the suites everyone uses, so the numbers mean something.
| Suite | Tutte 0.0.0 (frozen) |
|---|
| MMLU | 24.89% |
| HellaSwag | 25.63% |
| ARC-Challenge | 25.68% |
| GSM8K | 0.08% |
| HumanEval | 1.83% |
Tutte 0.0.1 Beta scores: pending — both models run these suites via our API, so every number is comparable.
Fig. 2 — internal synthetic suite, 7,176 items
Internal synthetic suite
Our own training benchmark. Attempt 2’s official rework bench (Sept 24) is the run with the full per-suite breakdown — frozen 0.0.0 seed against Tutte 0.0.1 beta, reported exactly as run.
| Skill | 0.0.0 seed | 0.0.1 beta · attempt 2 |
|---|
| Overall, weighted | 29.28% | 65.23% |
| Arithmetic, two operands | 71.1% | 100% |
| Tool calling | 99.4% | 100% |
| JSON field extraction | 94.5% | 96.9% |
| Arithmetic, three operands | 0.0% | 93.2% |
| Arithmetic, multiplication | 0.0% | 98.9% |
| Copy / repetition | 0.0% | 61.5% |
| Needle retrieval | 0.0% | 0% |
| Refusal calibration — below our 60% bar | 75.0% | 8.3% |
Refusal is the known weak spot: the beta over-answers where it should decline — 8.3% in attempt 2, 16.7% in attempt 3, against our 60% bar. The ledger below tells the whole story.
Fig. 3 — the attempt ledger
Four attempts, one bar
Every 0.0.1 run measured against the frozen seed. Our release bar: beat 29.28% overall, move the baseline-zero suites, hold tool calling above 90%, and clear 60% refusal calibration — the gate no attempt has cleared yet. That is why nothing has shipped.
Attempt 1Archived · Sept 23
97 datasets, strictly sequential, no replay. 4.68% — catastrophic forgetting in action: JSON extraction 94.5%→0%, refusal 75%→0%. Archived as the failure that taught us the replay trainer.
4.68%internal suite
Attempt 2Official bench · Sept 24
The rework: family replay, retention gates, saturated waivers. 65.23% — 2.2× the baseline — with arithmetic and tool calling at 100%. No-go: refusal 8.3% against the 60% bar.
65.23%internal suite
Attempt 3Best official · Sept 24
Heavier replay, ramp protocol, 97 of 97 sets. 67.95% on the full 7,176-item bench, checkpoint hash-matched to the results. Refusal climbed to 16.7% — real progress, still short of 60%. No-go.
67.95%internal suite
Attempt 4Superseded · Sept 25
Refusal-focused repair: probes reached 75%, but a sandbox rollback wiped the run’s artifacts. Rather than rebuild on sand, we started a fresh retrain with proper logging.
75%refusal probes
Current runIn the lab · now
Fresh retrain from the frozen seed, full logging, official like-for-like bench running. Scorecard pending. It ships only when every gate — refusal included — passes.
Pendingscorecard
We publish the 4.68% next to the 67.95% because a lab that hides its misses can’t be trusted with its hits. Attempts 1–3 are official benchmark runs recorded in the training log; the run artifacts were later lost to a sandbox rollback, so we report them as historical results rather than current releases. Attempt 4’s 75% is probe data, not a bench — marked as what it is.
The research thesis
Don’t outspend the frontier. Out-learn it.
We will never have the frontier labs’ compute, and we’ve stopped pretending otherwise. Our thesis is different: watch what the frontier discovers, adapt it fast, and automate the loop — AI agents proposing experiments, training candidates, benchmarking them, learning from the misses, with humans holding the promotion gate.
Tutte 0.0.0 was the control: an architecture, deliberately untrained. 0.0.1 is the first data point. The number we care about isn’t just the score — it’s the slope.
Small models are the point. A tiny model can be iterated thousands of times; a giant one can’t. Being compute-poor forced us to ask the more interesting question: how much intelligence can you extract from the least compute?
We optimizeΔ capability / training $·benchmark / parameter·improvement velocity
One model. An honest history.
No alphabet soup of tiers — one model, versioned in the open. Every release stays available for comparison. Nothing is quietly retired.
Tutte 0.0.1Beta · Gated
The strongest Tutte yet — 67.95% on the internal suite (attempt 3) — still gated on refusal calibration. It stays beta until every gate passes. Console preview live now.
67.95%internal suite
Tutte 0.0.0Frozen · Reference
The original 1.1M-parameter mixture-of-experts seed — pinned permanently as the baseline every future Tutte is measured against. 29.28% on the internal suite. Still served for comparison.
Next runIn the lab · now
Fresh retrain from the frozen seed — full logging, official like-for-like bench running. Scorecard pending. Ships only when every gate, refusal included, passes.
Every Tutte runs in the console with four modes: Mini for quick chats, Frontier for maximum reasoning, Uncensored for unfiltered conversation, and Era — set a year, and Tutte answers with the knowledge and voice of that time.
