Measured models · report cards
A model is more than one number.
rank 1 · 1500 trials
rustlean
41.5%
rank 2 · 1500 trials
lfm2.5-2.6b-fable5-coding-agent-heretic
34.3%
rank 3 · 1500 trials
minicpm5-2b-claude-fable5-1-thinking-agentic
27.8%
rank 4 · 1500 trials
minicpm5-1b-claude-opus-fable5-v2-thinking
18.1%
Selected model · 1500/1500 current trials
rustlean
kyber_rustlean.jsonl
Pass@1
41.5%
622/1500 · Wilson 39%–44%
Clustered
50.5%
891 answer shapes · Wilson 47%–54%
Grade weighted
0.474
valid outcomes, difficulty weighted
Mean score
0.432
41.5% valid-pass
Capability map
Pass@1 · each family's own denominator
Select
Tool selection
59.4%
101/170 first-answer passes · open trials
Args
Argument filling
77.6%
132/170 first-answer passes · open trials
Flows
Multi-step flows
61.3%
92/150 first-answer passes · open trials
Shell
Terminal skill
24.3%
33/136 first-answer passes · open trials
Files
File operations
39.5%
45/114 first-answer passes · open trials
Classify
Classifier judgments
41.6%
52/125 first-answer passes · open trials
Robust
Robustness
39.2%
49/125 first-answer passes · open trials
Omarchy
Omarchy control
10.2%
17/166 first-answer passes · open trials
Arch
Arch system care
0.0%
0/125 first-answer passes · open trials
Shell+
Pro shell tools
7.0%
8/115 first-answer passes · open trials
Keys
Hotkey knowledge
89.4%
93/104 first-answer passes · open trials
Grade cliff
Pass@1 across difficulty grades assigned by work, args, and shell composition.
Cost and outcomes
Model time
180ms
Tokens out
32
Peak heap
16.6 KB
Outcome kinds