Benchmarks

Generated by npm run bench:page from the run of 2026-10-27 at commit 440f18f. Do not edit by hand.

Published claims, settledCopy Anchor Link

Every number this repository states in public, checked against the record that settles it. "Not met" and "unmeasured" are different outcomes and never collapse into each other: one says we measured and it is not true, the other says we have not measured.

The two agent rows changed measure on 2026-10-09 (D-20261009-b1-totals-and-explain). This run settled them on the ratio of pooled medians, which commander's bimodal median turned into a coin flip. They are now settled on the tokens and turns each build spends summed over every task-run. Summed from this run's per-task detail, burgee ÷ commander is 0.692 for tokens (claim ≤ 0.6) and 0.703 for turns (claim ≤ 0.7). The first published run that carries the total records settles both rows here.

B2 — cold startCopy Anchor Link

Spawned processes, not an in-process micro-benchmark: process time is what a user and an agent actually pay, and excluding Node's own startup would flatter every row. bare node is the floor row and most of every other number.

These milliseconds are a property of the machine that ran them — AMD EPYC 7763 64-Core Processor, 4 cores, linux/x64, Node v24.13.0, on CI — and are not comparable across machines. Nothing gates on them. What is gated and banded is the ratio below, taken between two spawns interleaved in the same run, which cancels the machine out.

B1, the half that needs no model — what an agent can act onCopy Anchor Link

Not B1, and not a stand-in for it: this measures nothing about tokens or turns. It measures whether the CLI's answer is legible to an agent, which needs no model at all. Ten tasks per variant, one spawn each, non-TTY with stdin closed — the only environment an agent gets. The same demo program, built on each engine.

Exit code is the one that decides an agent's next move: 2 means rewrite the command, any other non-zero means the command was fine and the world was not. Both incumbents answer 1 to a usage error, which tells an agent nothing — so it retries a malformed command until it gives up.

Bytes is reported against us and is not gated. commander reads fewer than we do, because our errors carry a hint naming the fix. That is a trade — bytes per failure against failed turns — and only B1 proper can settle it. It is on this page precisely so the trade is visible rather than quietly omitted.

N10 — the floor, graded by someone else's checklistCopy Anchor Link

Every other number on this page comes from an instrument this repository wrote, and an instrument its authors wrote can only find the faults they thought of. This one hands the same demo program to two published, third-party checkers and records what they say: clispec 0.3.0, the scorer for The CLI Spec v0.3 (six principles, 24 points), and cli-agent-lint 0.3.5 (34 checks, graded A–F). Both are pinned by version and archive digest in benchmarks/floor-tools.ts, run offline, and are never re-scored here — a check they fail is listed as failed, even where we would argue with it.

Measured at commit 28843a5 on 2026-10-19.

What clispec 0.3.0 failed on burgee's demo:

• none

What cli-agent-lint 0.3.5 did not pass on burgee's demo (warn and info included; skip is not):

• none

B5 — runtime against the incumbentCopy Anchor Link

"Faster" is a claim like any other, so it is measured like one. One realistic workload per pair — the mixed text a CLI prints, a redrawn frame, a 20-row wrapped table, a prompt typed into and submitted — each in its own process, in front of a TTY with colour and hyperlinks forced, and each first proving that ours and the incumbent print the same bytes: a ratio between two different jobs is the flattering number this page exists to refuse. Eleven interleaved rounds; the number is the median of the per-round ratio, ours ÷ theirs, so a round's load lands on both. A reading over its ceiling is confirmed by two more fresh processes before it fails, so load on one process cannot fail a pull request and a regression in every process still does.

The target is ≤ 1.0 for every pair. Where a pair is above it, the gate is a ceiling just above its CI measurement that may only go down: the PR that makes the pair faster lowers it, and raising one takes a recorded decision (.sdlc/bands/runtime-ratchets.json).

Measured at commit 440f18f:

The ceilings every pull request is gated on today:

B3 — compatibilityCopy Anchor Link

Read from compat-oracle and never recomputed here: two implementations of a compatibility rate produce two rates. The full method, and the control column that proves the gate works, are on Compatibility.

B4 — weightCopy Anchor Link

Bundled is what your application grows by when you import that entry point: a fixture importing exactly it, one symbol used, esbuild-minified — the incumbent measured by the same command over the same fixture shape. A tarball size answers a different question.

Installed is what npm install puts on disk, including everything the package drags in. Ours drag nothing, which is where the two columns diverge most; but one of our packages serves every entry point it publishes, so the installed column is not per-entry-point and nothing gates on it.

A complete CLI — the dependency billCopy Anchor Link

What npm install puts on disk for a CLI that parses its arguments, colours its output, shows a spinner, prints a table and asks a question. Ours is the union of each package installed alone, offline, from its tarball (supply chain). Theirs is each incumbent's tree as package-lock.json resolves it, and each tree's size agrees with the one the upstream watch resolved from the registry for the same release. A package installed at two versions counts twice: both are on disk.

Paired by job: parse arguments (burgee for commander), colour (roundel for chalk), spinner (flagstaff for ora), table (flagstaff for cli-table3), prompts (caique for @inquirer/prompts).

Ours installs bellpull, burgee, caique, closeout, flagstaff, linegauge, paratext, roundel, seniority, every one published from this repository. Theirs installs ansi-regex, string-width, strip-ansi twice, at two versions each; the incumbents are separate projects, so the last column has no meaning for them.

ReproduceCopy Anchor Link

npm install
npx turbo run build
npm run bench -- --check

Results are written to benchmarks/results/<suite>/<date>.json and read by the Stage 6 control bands in .sdlc/bands/control-bands.json. The suite refuses to emit a band value it did not measure; see benchmarks/README.md.