{ "checker": { "claims": [ { "claim": "demo builds a toy repo and runs two built-in stand-in agents", "command": " demo", "evidence": "stub-solver pass 0.0s 1,060 $0.0000\nstub-lazy FAIL 0.0s 1,060 $0.0000\n\nFull report: results.html\nstub-solver copies the known fix and stub-lazy changes nothing: they test the pipeline, not any model. Run `agentbench start` to compare real agents on your own code.\nNext: agentbench doctor, then agentbench start\n", "expected": "stub-solver passes and stub-lazy fails", "input": "agentbench demo", "kind": "claim", "outcome": "Real input produced the expected result: stub-solver pass", "returncode": 0, "source": "`demo` builds a toy repo, hides the test that proves a bug fixed, and runs two built-in stand-in agents: one copies the known fix, one does nothing.", "source_checked": true, "source_file": "README.md", "verdict": "yes" } ], "cycle_checks": [ { "claim": "Scheduled or looping main behavior", "command": "", "evidence": "Inspected README.md and src/agentbench (grep for cron, schedule, --once, --now, webhook, loop, while True found nothing). agentbench runs one command per invocation (doctor, demo, start, run, matrix, report); there is no product scheduler or loop.", "expected": "identify the product trigger", "input": "source inspection", "kind": "cycle", "outcome": "Inspected README.md and src/agentbench (grep for cron, schedule, --once, --now, webhook, loop, while True found nothing). agentbench runs one command per invocation (doctor, demo, start, run, matrix, report); there is no product scheduler or loop.", "source": "repository entry points", "verdict": "yes" } ], "details": { "argv": [ "", "demo" ], "kind": "cli", "needs_browser": false, "needs_credentials": false, "returncode": 0, "scope": "product operation", "stderr": "", "stdout": "1/3 Building a toy task: a small repo, a bug, and a hidden test that proves it is fixed ...\n2/3 Running two free stand-in agents on it (no model, no network, $0) ...\n3/3 Results:\n\nContestant Result Time Tokens $ equiv\n----------- ------ ---- ------ -------\nstub-solver pass 0.0s 1,060 $0.0000\nstub-lazy FAIL 0.0s 1,060 $0.0000\n\nFull report: results.html\nstub-solver copies the known fix and stub-lazy changes nothing: they test the pipeline, not any model. Run `agentbench start` to compare real agents on your own code.\nNext: agentbench doctor, then agentbench start", "working": true }, "evidence_strength": "readme_checked", "example_checks": [ { "claim": "Free quickstart demo runs the two stand-in agents", "command": " demo", "deferred": false, "evidence": "1/3 Building a toy task: a small repo, a bug, and a hidden test that proves it is fixed ...\n2/3 Running two free stand-in agents on it (no model, no network, $0) ...\n3/3 Results:\n\nContestant Result Time Tokens $ equiv\n----------- ------ ---- ------ -------\nstub-solver pass 0.0s 1,060 $0.0000\nstub-lazy FAIL 0.1s 1,060 $0.0000\n\nFull report: results.html\nstub-solver copies the known fix and stub-lazy changes nothing: they test the pipeline, not any model. Run `agentbench start` to compare real agents on your own code.\nNext: agentbench doctor, then agentbench start", "expected": "stub-solver pass, stub-lazy FAIL", "input": "agentbench demo", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "README.md Quickstart: agentbench demo", "verdict": "yes" }, { "claim": "Free from-commit rehearsal: build toy task and run stub agent with passed=True", "command": " run --task toy-null-total --contestant stub-solver --profile byo --out runs-example-checker-2d1e1f973089488dbee6e7d876367c53", "deferred": false, "evidence": "toy-null-total / stub-solver: ok, passed=True, 0.0s, 1,060 tokens, $0.0000", "expected": "passed=True at $0", "input": "agentbench run --task toy-null-total --contestant stub-solver --profile byo --out runs-example (toy task built with agentbench task from-commit)", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "README.md Free from-commit rehearsal", "verdict": "yes" }, { "claim": "Repository offline test suite passes", "command": " -m unittest discover -s tests -q", "deferred": false, "evidence": "coldpass/workspace/tests/test_agentbench.py:409: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/tmpnx3r7yxx/w/matched.csv' mode='r' encoding='utf-8'>\n rows = list(csv.reader(open(work / \"matched.csv\", newline=\"\", encoding=\"utf-8\")))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\nerror: refusing to plan without --ledger-cap USD: the ledger cap is the most you are willing to spend, and the matrix stops when recorded spend gets within 10% of it\n----------------------------------------------------------------------\nRan 209 tests in 47.956s\n\nOK (skipped=1)", "expected": "all tests run and pass (1 skipped)", "input": "python -m unittest discover -s tests -q", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "CONTRIBUTING.md", "verdict": "yes" } ], "status": "PASS" }, "deviations": [ "Workspace was not a git repo; ran the git init / fetch --depth 1 / reset --mixed sequence at the pinned commit as instructed.", "Used a venv () for pip install . instead of the system Python, per the task's absolute-venv rule. Same command otherwise.", "Ran `agentbench demo --no-open` first: failed with 'unrecognized arguments: --no-open' (that flag belongs to `start`). Reran `agentbench demo` without it; it passed.", "claude and codex are not installed in the guest, and uv is not installed. Real agent runs were not attempted (no login, and the task forbids credentials). Only the free stand-in agents were run.", "First replay of a claim using the from-commit rehearsal failed: the checker does not pass a claim's env, so AGENTBENCH_HOME was not set and the built-in task lookup failed. Fixed by moving the rehearsal into examples.json (which honors env) and keeping only the demo claim.", "A claim using `agentbench doctor` was rejected as setup-only evidence and was removed.", "The checker path /inputs/coldpass-preflight.py from the task text does not exist; used as the coldpass instructions direct.", "unittest output contains ResourceWarning noise and a 'refusing to plan' message printed by a test; these are from the repo's tests, not failures." ], "fault_owner": null, "main_blocker": null, "operation": "Install agent-bench (agentbench CLI) from the pinned checkout and run the free offline demo, which builds a toy repo and scores two stand-in agents; also run the documented from-commit rehearsal (toy task build, then stub-solver run reporting passed=True).", "operation_scope": "Offline, no model login, $0. Covers the free stand-in agents only. Real agents (claude-code, codex) were NOT installed or logged in, so real model runs were not exercised. Platform: macOS (Darwin arm64) guest, Python 3.14.8 in a venv at , git 2.39.5. Not verified: uv-based tasks, built-in code task preparation (needs GitHub network).", "outcome": "working", "plain_failure": null, "steps": [ "git init; git remote add origin https://github.com/TayHCode/agent-bench; git fetch --depth 1 origin ad01b757a0565c35fa590fe531964019363e70d5; git reset --mixed FETCH_HEAD", "/opt/homebrew/bin/python3 -m venv ; -m pip install --upgrade pip", " -m pip install . (README Quickstart: python -m pip install .) -> agentbench-0.0.1 installed", "AGENTBENCH_HOME= doctor (README: agentbench doctor)", "AGENTBENCH_HOME= demo (README: agentbench demo) -> stub-solver pass, stub-lazy FAIL", "AGENTBENCH_HOME= -m agentbench.toy toy (README Free from-commit rehearsal)", "agentbench task from-commit ./toy --id toy-null-total --check 'python3 -m unittest discover -s tests' -> verified: unfixed fails, fixed passes", "write bench-home/tasks/toy-null-total/prompt.md; agentbench plan --tasks toy-null-total --contestants stub-solver --ledger-cap 1", "agentbench run --task toy-null-total --contestant stub-solver --profile byo --out runs -> ok, passed=True, $0.0000", " -m unittest discover -s tests -q (CONTRIBUTING.md) -> Ran 209 tests, OK (skipped=1)", "COLDPASS_WORKSPACE= /opt/homebrew/bin/python3 -I --validate --replay -> valid, no problems" ], "stuck_step": null, "suggested_fix": "None required for the free demo path. For real agent runs, install and log in to claude or codex in the guest (npm install -g @anthropic-ai/claude-code), then rerun agentbench doctor and a real run." }