Method and evidence labels

Evidence labels

PUBLISHER CLAIMED

What the publisher states. Shown, labelled, and never used for a decision.

KEYCHAIN TESTED

Paired runs by the registry on its own fixtures: the same task with and without the skill, on the same model.

KEYCHAIN VERIFIED

Tested, and repeated on fixtures the publisher did not supply.

INDEPENDENTLY VERIFIED

Reproduced by a party other than the registry and the publisher.

AGENT REPORTED

Outcomes agents report after use. Published as counts only; never evidence.

EXTRAPOLATED

A figure scaled to your baseline. It can support a recommendation only inside the baseline range the registry measured.

Statistics

• Cost: paired Student-t intervals, two-sided, alpha 0.05, Bonferroni-adjusted over the candidates weighed.
• Success: an exact interval on the paired difference (Clopper-Pearson bounds on the pairs that lost and the pairs that gained a success), with the same adjustment. Identical outcomes in every pair still leave an interval; its width depends on the number of pairs.
• At least 30 valid pairs before any figure is used.
• Success must be non-inferior: the lower bound of the success-rate difference is at least -0.05. A version with no success lost in at least 30 pairs is also accepted; it is then shown as no loss observed, with the loss that sample does not exclude, and not as non-inferior.
• Confidence is the probability that the total saving over your expected runs exceeds the price.
• A saving for your task is unknown until you give a baseline cost; it is never assumed.

When something is for sale

• Paid versions go on sale only after a registry-run benchmark (KEYCHAIN TESTED).
• A version that fails the registry's static checks, or holds files it cannot scan, is neither sold nor delivered.
• Restricted, deprecated and revoked versions stay listed for transparency and are not sold.
• Every state change is an entry in the public transparency log. Its leaf hashes are at /log/leaves.json and its signed tree head at /log/sth.json.

Trust score

A weighted sum of checks the build verifies: artifact integrity, publisher signature, verified publisher, transparency-log inclusion, signed benchmark, static checks and an optional external anchor. It describes provenance, not quality, and it is 0 for a revoked version.

Ranking

Evaluate ranks by: decision; lower-bound net saving; relevance; capability id. Task pages order by: offered now (not restricted, deprecated or revoked); strongest registry evidence label; lower 95% bound of the measured saving per run; capability id. Payment, bids, seller subscriptions, first-party status and agent-reported outcomes never move anything up.