Head to head · paired evidence

Two models. Same trials.

First-answer outcomes align by trial id. Wins count only where one model passes and the other misses; McNemar tests the discordant pairs. Partial overlap is reported as paired coverage, never hidden in two separate denominators.

Model AModel B

Different elicitation settings

The response protocol or decoding settings differ. Read the paired outcomes as a comparison of these recorded runs; the model and elicitation effects are combined.

A contract B contract

A pass@1 · paired

34.3%

188 exclusive wins

B pass@1 · paired

41.5%

295 exclusive wins

Shared trials

1500

327 both pass · 690 both miss

McNemar χ²

23.70

signal at p<0.05

Paired verdict

rustlean leads on paired passes.

188 A-only wins · 295 B-only wins · 1500 matched trials. The test uses Yates correction below 25 discordant pairs, matching the Rust engine.

Open A report Open B report

Where they separate

Percentage points · paired within family

Multi-step flows

-42.0 pp

150 paired trials

19%

61%

Hotkey knowledge

-30.8 pp

104 paired trials

59%

89%

Argument filling

-29.4 pp

170 paired trials

48%

78%

Classifier judgments

+21.6 pp

125 paired trials

63%

42%

File operations

+14.0 pp

114 paired trials

54%

39%

Omarchy control

-6.6 pp

166 paired trials

10%

Arch system care

+6.4 pp

125 paired trials

Tool selection

+5.3 pp

170 paired trials

65%

59%

Terminal skill

-3.7 pp

136 paired trials

21%

24%

Pro shell tools

-3.5 pp

115 paired trials

Robustness

-1.6 pp

125 paired trials

38%

39%

Proof, trial by trial

Discordant trials