{ "checker": { "claims": [ { "claim": "The demo exercises the pipeline with two free stand-in agents and demonstrates the known fix passes while doing nothing fails.", "command": " demo --html ", "evidence": "agent-bench demo

agent-bench demo

2 coding-agent setups (model + CLI) on 1 checkable tasks. Dollar figures are dollar-equivalents at list price (2026-10-09 table) from subscription runs. Read \u201cMethodology and limits\u201d before quoting any number.

2runs recorded
1passed the hidden check
0killed at a cap
$0.00dollar-equivalent, all runs

Conclusion

No conclusion text supplied.

Leaderboards

Passed = finished inside every cap and passed the hidden check. Work passes check = the hidden check passed on what the run had written, even when it was killed at a cap. Ranked by share of runs passed, then dollars per solved run (all spend, failures included, divided by passes), then wall time. Runs held to different caps are ranked separately; the task sets differ too (the M4 rerun covers only the three harder tasks), so the two boards are not comparable with each other.

Your runs: cap on fresh tokens, dollars and minutes (see the Cap profile column)

#ContestantPassedWork passes checkTasks all-pass$ spent$ per solved runMean wall sMean total tokensMean fresh tokensWorkersCappedFinished or capped
1stub-solver1/11/11/1$0.000$0.00001,0601,0601 finished / 0 capped
2stub-lazy0/10/10/1$0.000n/a01,0601,0601 finished / 0 capped

Cost against outcome

passedfailed or capped
$0.01$0.1$1stub-solvertoy-null-total (byo): pass, $0.000, 0 sstub-lazytoy-null-total (byo): fail, $0.000, 0 s
One mark per run, log-scale dollar-equivalent. Within a row the vertical slot is the task, top to bottom: toy-null-total. Hover or long-press a mark for its numbers.

Cost per solved run

stub-solver$0.000 \u00b7 1/1 passed
stub-lazyno solved run

All spend on a contestant (failed and capped runs included) divided by its passes. Linear scale.

Per task

toy-null-total

total() crashes on an empty cart

ContestantCap profilePassedWork passes checkMean $$ rangeMean wall sWall rangeMean total tokensMean fresh tokensWorkersCappedFinished or capped
stub-solverbyo1/11/1$0.00001,0601,0601 finished / 0 capped
stub-lazybyo0/10/1$0.00001,0601,0601 finished / 0 capped

Variance between repeats

Every cell is a single run, so there is no variance estimate.

Failed and capped runs

TaskContestantCap profileResultFinished or cappedWhat the output looked like
toy-null-totalstub-lazybyofailfinishedfinished and stopped, but the hidden check failed (0 files edited (+0/-0)): FAILED (errors=1)

Passing diffs against the real fix

Source lines changed (added + removed, tests excluded) by each passing run, against the upstream pull request. A size comparison, not a quality verdict; test files, docs and change logs, lockfiles an install wrote, and files a task manifest lists as unrelated to the fix (three type-annotation files in the taskiq pull request) are left out. Data tasks have no code fix to compare.

TaskContestantCap profileRun linesFix linesRatioFix files touchedVerdict
toy-null-totalstub-solverbyo441.0\u00d71/1near the real fix

All runs

TaskContestantCap profileResultWall sTotal tokensFresh tokens$ equivWorkersEffort seenCappedFinished or capped
toy-null-totalstub-lazybyo: 100k fresh tokens, 2 min, $0.01fail01,0601,060$0.0000?nofinished
toy-null-totalstub-solverbyo: 100k fresh tokens, 2 min, $0.01pass01,0601,060$0.0000?nofinished

Workflow-match check

Five descriptive questions per task document whether it resembles real work. These questions are not a scoring gate.

Methodology and limits

  • The unit is model + CLI, not the model alone. Each column bundles a model with that vendor\u2019s agent harness and any config files you listed for it.
  • Small sample. 1 task(s) of your own, 1 run per cell. A pass means the run worked once, not that the contestant is reliable. Treat this as a first look at your own work, not a ranking.
  • Settings held equal: the same effort level for every contestant unless you set otherwise, connectors off, a fresh checkout and a clean HOME per run, the same prompt, one process at a time. Per-run caps on fresh tokens, dollars and minutes are enforced; a run that hits one is killed, marked capped and counts as not passed.
  • Pass means the run finished inside every cap and the hidden check passed. The hidden check is copied in only after the run. It is", "kind": "cli", "needs_browser": false, "needs_credentials": false, "returncode": 0, "scope": "product operation", "stderr": "", "stdout": "1/3 Building a toy task: a small repo, a bug, and a hidden test that proves it is fixed ...\n2/3 Running two free stand-in agents on it (no model, no network, $0) ...\n3/3 Results:\n\nContestant Result Time Tokens $ equiv\n----------- ------ ---- ------ -------\nstub-solver pass 0.1s 1,060 $0.0000\nstub-lazy FAIL 0.0s 1,060 $0.0000\n\nFull report: \nstub-solver copies the known fix and stub-lazy changes nothing: they test the pipeline, not any model. Run `agentbench start` to compare real agents on your own code.\nNext: agentbench doctor, then agentbench start", "working": true }, "evidence_strength": "readme_checked", "example_checks": [ { "claim": "README quickstart demo", "command": " demo --html ", "deferred": false, "evidence": "agent runs as your own user, so a determined agent could look for it. Runs whose transcript shows that (the peeked flag) are left out of every table.
  • Dollars are dollar-equivalents. Tokens \u00d7 the dated public list prices (2026-10-09); a subscription is not billed per token but uses plan limits. It is not an invoice.
  • Privacy. Contestant CLIs send the files they read to their vendors. The runner has no telemetry or uploads. Explicit task preparation downloads pinned public sources; task setup and checks may download dependencies.
", "expected": "stub-solver passes, stub-lazy fails and HTML report is written", "input": "demo --html fresh file", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "README.md / Quickstart: three commands, no login, no cost", "verdict": "yes" } ], "status": "PASS" }, "deviations": [ "System pip install was attempted and rejected by PEP 668; installation succeeded in the README-recommended virtual environment." ], "fault_owner": null, "main_blocker": null, "operation": "Run the documented agentbench demo", "operation_scope": "Quickstart install and free demo on Python 3.14.8 in a project virtual environment", "outcome": "working", "plain_failure": null, "steps": [ "Created .venv with /opt/homebrew/bin/python3", "Installed the checkout with .venv/bin/python -m pip install .", "Ran .venv/bin/agentbench demo --html results.html", "Replayed the demo and README claim with the coldpass checker" ], "stuck_step": null, "suggested_fix": null }