Head to head · paired evidence

Two models. Same trials.

First-answer outcomes align by trial id. Wins count only where one model passes and the other misses; McNemar tests the discordant pairs. Partial overlap is reported as paired coverage, never hidden in two separate denominators.

Model AModel B

Different elicitation settings

The response protocol or decoding settings differ. Read the paired outcomes as a comparison of these recorded runs; the model and elicitation effects are combined.

A contract B contract

A pass@1 · paired

27.8%

159 exclusive wins

B pass@1 · paired

41.5%

364 exclusive wins

Shared trials

1500

258 both pass · 719 both miss

McNemar χ²

80.35

signal at p<0.05

Paired verdict

rustlean leads on paired passes.

159 A-only wins · 364 B-only wins · 1500 matched trials. The test uses Yates correction below 25 discordant pairs, matching the Rust engine.

Open A report Open B report

Where they separate

Percentage points · paired within family

Argument filling

-48.2 pp

170 paired trials

29%

78%

Multi-step flows

-34.0 pp

150 paired trials

27%

61%

File operations

-29.8 pp

114 paired trials

10%

39%

Hotkey knowledge

-26.0 pp

104 paired trials

63%

89%

Classifier judgments

+25.6 pp

125 paired trials

67%

42%

Tool selection

-21.8 pp

170 paired trials

38%

59%

Terminal skill

-16.2 pp

136 paired trials

24%

Robustness

+14.4 pp

125 paired trials

54%

39%

Arch system care

+5.6 pp

125 paired trials

Pro shell tools

-3.5 pp

115 paired trials

Omarchy control

-3.0 pp

166 paired trials

10%

Proof, trial by trial

Discordant trials