TayHCode/agent-benchWorksBenchmark AI coding agents on real, checkable work, or on y…Linux ✓ · macOS ✓ · Windows ?Maintained · last commit Oct 15Setup 47 sExpand checksCollapse checks
- What it does
- Benchmark AI coding agents on real, checkable work, or on y…
- Does it work?
- Works
- Runs
- 5 runs: passed 4 of 5
- Does it do what it says?
- 1/1 claims ok
- Is it safe?
- Nothing unexpected
- Is it maintained?
- Yes, last commit Oct 15
- Setup time
- 47 s
- Operating systems
- Linux ✓ · macOS ✓ · Windows ?
- Models tested with
- Haiku ✓ · Luna ✓
- Needs a key or login?
- None seen
- MCP?
- No
- Local or hosted?
- Local
| Platform | Haiku | Luna |
|---|---|---|
| Linux | yes | yes |
| Windows | blocked: setup step | not tested |
| macOS | yes | yes |
More details
README claims checked out on 4 recorded setups. Untested combinations remain unknown.
Linux / Haiku 5.5: yes · Oct 21, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Agent-reported task: Install agentbench from the checkout and run the free stand-in benchmark: `agentbench demo`, plus a from-commit toy task built and run with the stub-solver contestant.
Linux / Luna: yes · Oct 22, 2026 · ad01b757a056 · codex / codex-cli 0.163.0
Agent-reported task: agentbench demo
Windows / Haiku 5.5: blocked: setup step · Oct 22, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Agent-reported task: Follow the agent-bench README quickstart (python -m pip install., agentbench doctor, agentbench demo) and run it
macOS / Haiku 5.5: yes · Oct 22, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Agent-reported task: Install agent-bench (agentbench CLI) from the pinned checkout and run the free offline demo, which builds a toy repo and scores two stand-in agents; also run the documented…
macOS / Luna: yes · Oct 22, 2026 · ad01b757a056 · codex / codex-cli 0.163.0
Agent-reported task: Run the documented agentbench demo
Tested Oct 22, 2026 on ad01b757a056
- Does what it says
- 1 of 1 claims checked out
- Safety
- No sign of malicious behavior found in these checks
- Maintained
- Last commit Oct 15, 2026 · Maintained
- MCP
- No
- Local or hosted
- local
- Needs keys or browser
- None seen
- Setup time
- 47 s
- Examples
- 1 of 1 ran as expected
- Evidence
- README behavior checked
- Agent interfaces
- CLI
- Tested with
- Codex / Luna
Claims: The demo exercises the pipeline with two free stand-in agents and demonstrates the known fix passes while doing nothing fails.: yes
Examples: README quickstart demo: yes
Oct 21, 2026 · Linux / Haiku 5.5 · passed
Oct 22, 2026 · macOS / Haiku 5.5 · passed
Oct 22, 2026 · Windows / Haiku 5.5 · blocked: setup step
Oct 22, 2026 · Linux / Luna · passed
Oct 22, 2026 · macOS / Luna · passed
No problems traced to this repository code in these checks; a short run cannot prove safety.
Full event, path and process lists for each observed setup:
- linux · Haiku 5.5 safety evidence →
- macos · Haiku 5.5 safety evidence →
- windows · Haiku 5.5 safety evidence →
- linux · Luna safety evidence →
- macos · Luna safety evidence →
Linux / Haiku: yes · tested Oct 21, 2026 on ad01b757a056 · The independent checker confirmed the recorded README operation.
Linux / Luna: yes · tested Oct 22, 2026 on ad01b757a056 · The independent checker confirmed the recorded README operation.
Windows / Haiku: blocked: setup step · tested Oct 22, 2026 on ad01b757a056 · cause: test · Windows guest lacks nested virtualization, so WSL 2 cannot run and agentbench cannot start.
macOS / Haiku: yes · tested Oct 22, 2026 on ad01b757a056 · The independent checker confirmed the recorded README operation.
macOS / Luna: yes · tested Oct 22, 2026 on ad01b757a056 · The independent checker confirmed the recorded README operation.