phronesis · laserbrain
Every number here was wrong first
Each threshold in laserbrain was set by a judgement and corrected by a measurement. The corrections were larger than the final values and more useful to know about: a gate that blocked 74.3% of correct answers tells you more than one that blocks 6.7%.
The way we found out was the same every time — point the thing at something real, then read what it said instead of counting it. Every run found a defect no test had caught, and the tests were green throughout, because a suite written by the person who wrote the code confirms the convention they had in mind rather than the one the world uses.
what is measured now
The corrections
Each row names the file that fails the build if the value drifts back, so this page can be checked rather than believed.
What blocks a generated answer
was quotations and figures both block — a number cannot be paraphrased
74.3% of correctly-grounded human summaries were blocked. 67 of 74 blocking findings were figures.
now quotations only. Paraphrase was never the question; whether a claim could be grounded in a source the checker was not handed is.
pinned · scripts/test-rag-gate.ts
What counts as a quotation
was any double-quoted span of three words or more
three of six sampled blocks were titles — "The Devolution Project", "Se Va el Caimán". Works are set in quotation marks by convention.
now Title Case excluded. Blocking fell 13.9% → 6.7%.
pinned · scripts/test-anchor.ts
How an endpoint is read from documentation
was a method followed by a path — GET /v1/widgets
zero endpoint claims from ten published READMEs. Every fixture used that form; almost no real documentation does.
now absolute URLs and curl invocations too, with backslash continuations joined first.
pinned · scripts/test-docs-drift.ts
When a documented status has drifted
was any status that is not the documented one
four separate false-positive classes — a 403 error envelope, an unauthenticated prober, a required query parameter, a 5xx read as a wrong document.
now one rule: a status is checkable only when the request this prober can make is the request the documentation describes.
pinned · scripts/test-docs-drift.ts
What a figure is
was any run of digits, with an optional leading minus
2026-06-21 split into three figures — 2026, -06, -21. Two invented from every date on the page.
now a date is one figure and stays a figure; a hyphen between digits is a range, not a minus.
pinned · scripts/test-anchor.ts
What a name is
was a capitalised word not at the start of a sentence
I've, I'd and I'll read as proper names; "Status: Ready" joined across a colon; UNDEFINED read as an organisation.
now contractions of pronouns dropped, runs broken at punctuation, all-caps over six letters treated as emphasis. Names still only flag — capitalisation is not an entity model.
pinned · scripts/test-anchor.ts
When two sentences say the same thing
was a 0.5 overlap threshold, with a test file that was never wired into the build
the threshold could be moved anywhere in (0, 1] without failing. Every fixture had overlap exactly 1.0.
now fixtures either side, at 0.429 and 0.714, and an assertion that they straddle it.
pinned · scripts/test-clarity.ts
A reference that updates toward what it measures stops measuring
Read together, the rows above are one rule found in seven places. Every one is a reference that had come to depend on the thing it was supposed to be measured against — a gate calibrated on answers it had itself selected, ground states drawn from the corpus under test, a threshold fitted to the outputs it was meant to judge. Point any of them at something from outside and the number moved, usually by more than the number itself.
That is one rule with a proof behind it, not a habit of carefulness. Update a reference toward what it observes, at any rate at all, and the reading converges to a constant that no longer depends on how far the system has gone. It becomes a measure of speed still wearing the label of a distance — and it does this without an error, a warning, or a failing test. Only a reference fixed before measurement begins reports displacement.
It is the same rule the checks here enforce on your systems, which is why the corrections are worth publishing rather than quietly fixing: the tool is the rule, applied outward. The argument in full, with the derivation and six times it was broken in working code, is in The Frozen Reference.
One finding that was not ours
Pointed at published OpenAPI specs from the APIs.guru directory, laserbrain docs found that Vercel’s spec declares /edge-config under https://api.vercel.com. That path answers 404. The route is live one segment up, at /v1/edge-config, which answers 403 — so the spec’s server and path do not compose to the real endpoint, and anyone following it literally gets a 404.
Two other findings from the same run were triaged and reported differently: a double slash in clever-cloud’s own path, which is real but cosmetic, and a 404 at redirection.io that is either a moved route or a private one hiding behind 404 — nothing observable from outside distinguishes them, so it is reported as something to read rather than a defect.
What it still cannot do
A faithful paraphrase scores badly. A fabrication assembled from the source’s own words scores perfectly. A real name attached to the wrong claim passes, and so does a verbatim quotation attributed to the wrong speaker — it asks whether the words are in the document, not whether that person said them.
Each of those is pinned as a passing test. None can be quietly fixed without the build failing until this page is rewritten, which is the only mechanism we have found that keeps a limitation stated after the person who found it moves on.
run it yourself
npx laserbrain-check try
No files, no key, no model. The evidence harnesses are in the repository as scripts/evidence-*.ts and are repeatable — every number on this page can be re-measured rather than taken.