Reproducibility

This is an archived measurement under rubric 8. The current instrument uses different measurement rules. These files do not validate current scores; do not compare them with a new live run.

docs.stripe.com · audited 2026-09-30 · AgentFit v3.40.2-19-g3ff4d31 · rubric v8 · Claude Opus 5

A score you cannot repeat is not a measurement. So here is the test we owe you: we audited the same site six times in one day — three times by asking a language model to apply a documentation rubric, three times by running AgentFit — and we published what we recorded, including the parts that are inconvenient for us.

Read the columns downwards, not across. The left rubric has 26 criteria; AgentFit has 28, and the two weight them differently — a machine-readable specification is worth 25 points on the left and 17 here. 57 is not "the right answer" and 78 is not "wrong". The claim on this page is narrower, and it is the only one the data supports: one of these two procedures returns the same number every time, and the other does not.

Where the runs disagreed

All three model runs did the work properly: they fetched live URLs, worked through the criteria one by one, and reported what they found. By our own count from the transcripts — which we did not publish — each run made between 36 and 47 tool calls, and none of them invented a fact. Take both of those as our reading of the runs, not as something the files below let you check. They still landed 15 points apart, and eleven of those points sit in a single category.

Two kinds of disagreement are mixed together there. The small ones — A5, B3, B5, B6, D1, D3, D6, a point here or there, in both directions — are what we would expect from three attentive readers of the same page: one found the SDK tabs in the DOM, the other two concluded they exist only inside a JavaScript payload. Whether that is ordinary measurement noise, three runs cannot tell you; that part is a guess. Only the D1 disagreement is visible in the published summaries — the rest comes from our own comparison of the three transcripts. The large disagreement is a different animal, and it is why this page exists.

The remaining criteria in category C scored exactly 10 points in each of the three runs. The entire eleven-point gap is two criteria.

C1a: the same facts, three different scores

Criterion C1a — from the model's 26-criterion rubric; AgentFit folds discovery and validity into a single C1 — asks whether the API has a machine-readable specification an agent can reach. All three runs established precisely the same two facts about docs.stripe.com:

• the conventional locations — /openapi.json and its siblings — return 404;
• a valid, complete OpenAPI specification is published in Stripe's GitHub repository.

Nobody was wrong. Nobody hallucinated. They scored it 0 of 8, 3 of 8 and 4 of 8.

The disagreement is not about the site. It is about a question the rubric never asked: does a specification count as published if it lives on somebody else's domain and takes one more hop to reach? That is a genuine fork with no obvious answer, and each run walked up to it alone, in the middle of a long task, with no rule to consult. Three runs, three policies.

From there it cascades. C1b scores the quality of the specification you found. The run that decided GitHub does not count had nothing to validate and scored 0 of 7. The two that fetched it scored 7 of 7 — and then reported a different number of paths, 414 and 136, because Stripe's repository holds more than one specification file and they picked different ones (spec3.json and spec3.yaml). Two runs agreed on the score for opposite reasons.

E5 — "are the terms of use stated?" — came out 3/3, 2/3 and 1/3. One run found a footer link to stripe.com/legal/ssa. One rejected that link and credited the Content-Signal directive in robots.txt instead. One looked for /terms, /legal and /privacy on the documentation host, got three 404s, and scored what it saw. Three defensible readings of one line of rubric.

The self-check varied too

The prompt ships with a calibration anchor: a table of previously measured sites on which Stripe sits at 56, measured 13 May 2026. It is there precisely to catch drift. That table is also why the site here is docs.stripe.com — Stripe was already in somebody else's calibration anchor before we ran anything, so the site was picked for us, and picked before the experiment.

Every run used it. Every run noticed its own deviation — +22, +24, +9. Every run then re-examined its criteria, reasoned about what might have changed since May, and concluded that the anchor was stale and its own number was the better one. Three times, independently, in the same direction, with three different numbers.

This is the part we find most instructive. It is not carelessness — it is what conscientious work looks like when the procedure still has a free parameter in it. A self-check whose verdict changes between runs is not a self-check.

The anchor happens to sit one point from AgentFit's 57. Read nothing into that: it is a hand measurement, on a different rubric, from a different month, and it is not evidence that 57 is the true score.

Why the code answers the same

Nothing clever. Every criterion is a fetch plus a rule, and both are written down before the audit starts.

D2 is the machine-learning one. Two small classifiers run inside the audit: one scores how realistic a code example is, the other judges endpoint-page completeness. They are ONNX graphs with frozen weights compiled into the binary. The same input gives the same tensor and the same float, every time. Determinism does not require avoiding models; it requires avoiding models whose output you cannot pin down.

C1 is the very fork the three runs walked differently, and AgentFit scores 0 of 8 on it. Our rule is not "GitHub does not count" — it is a fixed list of places to look: seventeen conventional paths on the host being audited, an RFC 9727 api-catalog link, and any specification link found on the homepage we fetched. On docs.stripe.com none of those paths returned a specification and that page carried no such link, so nothing was found. A rule that followed one more hop into a vendor's repository would not be less correct. Ours is simply fixed before the audit starts, applied identically to every site in the corpus, and its scoring published on the rubric page. That is the whole difference — not a better answer, a fixed one. How C1 is scored.

curl -fsS https://agentfit.dev/static/reproducibility/agentfit-run1.json -o agentfit-run1.json
shasum -a 256 agentfit-run1.json

What we are not claiming

• We are not claiming 57 is right. We do not publish a true score for docs.stripe.com, because nobody has one. We publish a procedure and its output. And reproducibility is a necessary property, not a sufficient one: echo 57 is perfectly reproducible too — what it is not is sensitive to the site. Whether ours is, /browse is the place to judge: the same procedure spreads real sites across the whole range.
• We are not comparing two numbers. Different rubrics: different criteria counts, different weights. The comparison is meaningful within each column, not between them.
• This is one site, one day, one model, three runs. It is a demonstration, not a study, and we are not promising to re-run it on a schedule.
• The model is Claude Opus 5 — the same model that helped build this page and a good part of this service. We are showing a variance we reproduced in the tool we use ourselves, every day; read it with that in mind. We are not saying language models are bad at this. We are saying we could not get one to return the same number twice.
• Determinism is a property of the procedure, not a promise about the internet. Our three runs left one machine on one day, and the site served German-language bodies to that address — visible in the B1 and E1 evidence, and one model run recorded the same thing. Change the vantage point and you change the input. Same input, same output is the entire guarantee.

What a model is genuinely good at here

None of this makes the model runs worthless — it makes them a different instrument. The diagnosis in each of the three runs is good on its own terms: why a missing specification hurts an agent, what to publish instead, which of Stripe's pages would survive being read without a browser. That is explanation, and a model explains beautifully.

A language model is excellent at explaining what to fix. It is a poor instrument for measuring whether you fixed it. We are about the second one. AgentFit is the instrument; take its number to a model and ask what to do about it. That is the right division of labour, and it is the one we use.

The data

All six runs are served from this site. The three AgentFit reports are the files the tool wrote, byte for byte — that is what the checksum is taken over. The three model files are ours:

The raw transcripts are not published. What you get for the model column are structured summaries we wrote afterwards from those transcripts: the final score, the category totals, and the criterion-level notes quoted above. The scores and totals are each run's own output; the wording of the notes is ours.

• three AgentFit reports and the checksum — run1.json · run2.json · run3.json · sha256.txt
• three model runs, as summaries we wrote from the transcripts — runA.json · runB.json · runC.json
• the prompt used for the model column, pinned at the commit we ran — ai-ready-audit

Nothing was filtered out — no run, and no site. Three model audits and three tool audits, on the one site the prompt's calibration table already named; we ran no others and discarded none. Had a fourth run come back at 57, we would have published that too, and this page would be shorter.

Audit your documentation →

Free, no sign-up, about 30 seconds. 28 criteria, HTTP evidence under every one, a shareable report link — and the same number tomorrow if nothing has changed.