Repo Cartographer benchmark evidence
Evidence
| Label | Model | Pairs | Saving per run (95% CI) | Cost reduction (median) | Success without → with | Fixtures |
|---|---|---|---|---|---|---|
KEYCHAIN TESTED | claude-sonnet-5 | 30 | USD -0.0269 [USD -0.0531, USD -0.0008] | -1% | 97% → 87% | synthetic |
Measured on the registry's fixtures. On this evidence the API answers do_not_recommend whatever your baseline: success with the capability is not shown to be at least as good (difference interval [-0.3321, 0.1641], margin 0.05).
Limitations the API states with these figures:
- The figures were measured on claude-sonnet-5; for another model they are for reference only.
- Benchmarked on synthetic fixtures with planted answers; treat the savings as an upper bound.
- Install and review time is not estimated.
KEYCHAIN TESTED · claude-sonnet-5 · 2026-09-25T13:08:30Z
- Suite
agentickeychain/repo-cartographer/synthetic-v1- Suite SHA-256
4d48582a3e896e067db08aec6a62ac33f556c67c1f365e6628a5cf8a3da2f133- Dataset SHA-256
b8744ea2e1cc2d403090ea086ba35f0eeb76a722d383af3e7b03433d1cc4490f- Evidence SHA-256
63bac5276fdf62aa059740b772c0aa5adbaf75a0eab9f00a9117ccebc01c118c- Valid pairs
- 30 (excluded 0)
- Capability runs that failed in the runner
- 0 (counted as failures)
- Pairs without a known cost
- 0 (left out of the cost figures)
- Mean cost without / with
- USD 0.1241 / USD 0.1510
- Baseline cost range
- USD 0.0627 – USD 0.3858
- Token reduction (median)
- 0%
- Tool-call reduction (median)
- 0%
- Skill actually used
- 3%