Log Triage benchmark evidence
Evidence
| Label | Model | Pairs | Saving per run (95% CI) | Cost reduction (median) | Success without → with | Fixtures |
|---|---|---|---|---|---|---|
KEYCHAIN TESTED | claude-sonnet-5 | 30 | USD 0.0895 [USD 0.0489, USD 0.1301] | 40% | 100% → 100% | synthetic |
Measured on the registry's fixtures. Your own saving depends on your baseline, which this page does not know, so it is unknown here; evaluate_capability computes it for your task and says whether the price is recovered.
Limitations the API states with these figures:
- The figures were measured on claude-sonnet-5; for another model they are for reference only.
- Benchmarked on synthetic fixtures with planted answers; treat the savings as an upper bound.
- No success was lost in 30 paired runs. That sample does not exclude a loss of up to 0.1359 in success rate; the 0.05 margin is not shown.
- Install and review time is not estimated.
KEYCHAIN TESTED · claude-sonnet-5 · 2026-09-25T13:07:05Z
- Suite
agentickeychain/log-triage/synthetic-v1- Suite SHA-256
db5e417f05853285d72951012f82bf1c698b0c0ac193fe4a19b3e482e1be1074- Dataset SHA-256
519d04fa16a4c95dad2a556f16cbf0314651fd8111dc799f68402ccf9a5c1593- Evidence SHA-256
69fe5b76c9689edbc8c6e301474e6b04567d71725466b745c8853d3c493140f1- Valid pairs
- 30 (excluded 0)
- Capability runs that failed in the runner
- 0 (counted as failures)
- Pairs without a known cost
- 0 (left out of the cost figures)
- Mean cost without / with
- USD 0.1979 / USD 0.1083
- Baseline cost range
- USD 0.0652 – USD 0.3735
- Token reduction (median)
- 42%
- Tool-call reduction (median)
- 45%
- Skill actually used
- 100%