AI coding agent repository benchmark
Implement one bounded feature in an existing repository and pass its acceptance tests.
Pilot the same bounded repository change with the tools that fit your approved development environment. Compare accepted code, review time, security boundaries and complete cost; this protocol does not establish the fastest or best coding tool.
The evidence table.
Weights describe the proposed evaluation. Unknown or test-pending evidence receives no score and no total. Until every tool has complete evidence, row order is an editorial shortlist—not a performance ranking. Approved user runs remain separate.
Fixed protocol.
- 01Use the same repository snapshot and written acceptance criteria.
- 02Allow the agent to inspect only the files it requests.
- 03Run the same automated tests and one human code review.
- 04Record accepted code, review minutes, retries and regressions.
What counts as usable.
✓ All stated acceptance tests pass
✓ No unrelated file changes
✓ Security boundaries remain intact
✓ A human reviewer can explain the implementation
Open comparison library →Approved same-task runs
Structured observations make the protocol challengeable. Every submission is moderated; supplier claims and editorial results remain separate.
No approved community runs yet. No completed comparative result is claimed; use the protocol to collect your own evidence.
Test one tool under the published rules.
Report completion, elapsed time, retries, correction time and cost. Upload an output screenshot only when you own it or have permission.
The editorial table is a transparent decision model built from HubAI profile data. Community runs are shown only after moderation and never presented as a controlled HubAI laboratory result.
Editorial disclosure →