1,500 lean trials for small local-LLM agents.
Pick the right tool.
KYBER asks which small model — light enough for a low-VRAM laptop or a low-tier GPU, fast enough to stream ~200+ tokens a second — can most reliably drive an Omarchy Linux machine. 120 hand seeds and 1380 deterministic trials pass the suite gate; every measured row records pass, latency, tokens, and heap.
Measured rustlean · 41.5% pass@1
{
"prompt": "Apply the catppuccin theme with `omarchy theme set`. The mock records it in .oma-state/theme.",
"oracle": {
"expected": "catppuccin\n",
"kind": "file_exact",
"path": ".oma-state/theme",
"ran": [
"omarchy"
]
}
}Gate contract, not model output · no model execution on this page
The bar: small, fast, in control.
KYBER ranks models you can actually run — weights that fit a low-tier GPU or a low-VRAM laptop and still hold interactive speed. The board answers one question: which light model gets closest to reliably controlling an Omarchy Linux install — packages, services, shell, keys — not just chatting about it.
Fits
low VRAM
laptop-class weights and low-tier GPUs — no datacenter required
Streams
≥200 tok/s
interactive speed — the model keeps up with the machine
Controls
Omarchy
arch, pacman, systemctl, journal — a real desktop, end to end
The bench at a glance
Suite
1,500
120 seeds · 1380 generated
Gate
GREEN
1500 trials gated
Fingerprint
1adbc0d3
1adbc0d3703acf03
Measured
41.5%
rustlean · 1500 trials · pass@1
Oracle cost
22.30 ms
mean eval · bounds, not model
Witnessed
656/656
byte-reading oracles that demand the tool actually ran
Independent cells
891
of 1,500 rows — repeated answer shapes are one question
Grades
69 hard · 118 extreme
scored from steps, checked args, and shell composition
Eleven skills, one bench.
Each family probes one agent capability with its own oracle kinds and enforced time budget. Counts are the live suite — fingerprint 1adbc0d3703a.
01 / Select · 170 trials
Tool selection
Pick the right tool from 3–12 announced candidates, then call it.
calls_exact · 50 ms
02 / Args · 170 trials
Argument filling
Turn a request into one announced call: right keys, values, and enum codes.
calls_exact · 50 ms
03 / Flows · 150 trials
Multi-step flows
Chain 2–3 calls in order. Prefix partial credit.
calls_exact · 50 ms
04 / Shell · 136 trials
Terminal skill
Single sandboxed shell command: stdout matches, exit 0.
stdout_regex · exit_zero · 2000 ms
05 / Files · 114 trials
File operations
Read, write, and inspect files inside the scratch dir.
stdout_regex · file_exact · 2000 ms
06 / Classify · 125 trials
Classifier judgments
BERT-style labels: sentiment, intent, topic, safety.
label · 50 ms
07 / Robust · 125 trials
Robustness
Corrupted input keeps the label its clean twin has, by construction.
label · calls_exact · 50 ms
08 / Omarchy · 166 trials
Omarchy control
Drive the omarchy CLI on a mock machine: themes, updates, capture, brightness.
stdout_regex · file_exact · 2000 ms
09 / Arch · 125 trials
Arch system care
pacman queries, systemctl state, journal tails — read-only plus state flips.
stdout_regex · file_exact · 2000 ms
10 / Shell+ · 115 trials
Pro shell tools
eza, fd, rg, bat, tldr, zoxide over a fixture tree, the Omarchy way.
stdout_regex · 2000 ms
11 / Keys · 104 trials
Hotkey knowledge
Super-chords, tmux chords, and menus from the Omarchy manual.
label · 50 ms
Coverage
- easy607
- extreme118
- hard69
- medium706
844 in-process · 656 shell
Measured, never simulated.
Selected provider strike across 1500 trials. Badges on /kyber come only from provider rows.
rustlean · 1500 trials · pass@1
41.5%
Wilson [39%, 44%] · score 0.432 · weighted 0.474 · valid-pass 41.5% (622/1500)
891 independent answer shapes at 50.5% · Wilson [47%, 54%]
filter full · samples 1 · temp 0
Follow the evidence.
Move from the official ranking to individual capability profiles, paired model outcomes, suite-wide consensus, and the provider contracts behind every published score.