Head to head · paired evidence

Two models. Same trials.

First-answer outcomes align by trial id. Wins count only where one model passes and the other misses; McNemar tests the discordant pairs. Partial overlap is reported as paired coverage, never hidden in two separate denominators.

Model AModel B

Different elicitation settings

The response protocol or decoding settings differ. Read the paired outcomes as a comparison of these recorded runs; the model and elicitation effects are combined.

A contract B contract

A pass@1 · paired

41.5%

295 exclusive wins

B pass@1 · paired

34.3%

188 exclusive wins

Shared trials

1500

327 both pass · 690 both miss

McNemar χ²

23.70

signal at p<0.05

Paired verdict

rustlean leads on paired passes.

295 A-only wins · 188 B-only wins · 1500 matched trials. The test uses Yates correction below 25 discordant pairs, matching the Rust engine.

Open A report Open B report

Where they separate

Percentage points · paired within family

Multi-step flows

+42.0 pp

150 paired trials

61%

19%

Hotkey knowledge

+30.8 pp

104 paired trials

89%

59%

Argument filling

+29.4 pp

170 paired trials

78%

48%

Classifier judgments

-21.6 pp

125 paired trials

42%

63%

File operations

-14.0 pp

114 paired trials

39%

54%

Omarchy control

+6.6 pp

166 paired trials

10%

Arch system care

-6.4 pp

125 paired trials

Tool selection

-5.3 pp

170 paired trials

59%

65%

Terminal skill

+3.7 pp

136 paired trials

24%

21%

Pro shell tools

+3.5 pp

115 paired trials

Robustness

+1.6 pp

125 paired trials

39%

38%

Proof, trial by trial

Discordant trials