Measuring confident error in rare disease reasoning

01 — THE DOMAINWhy I chose the hardest place to be right

A language model asked about a common condition is standing on a mountain of text. Thousands of papers, decades of guidelines, millions of described cases. It can lean on a strong statistical prior and be right for reasons that have nothing to do with reasoning.

Rare disease takes that away. For most of these conditions the published literature is a handful of case reports, sometimes a single family. The prior is gone, and what is left is whatever the model actually does when it does not know.

That is the interesting question, and it is not an academic one. These are the numbers I keep coming back to.

• 7,000+rare diseases described so far
• ~300Mpeople living with one worldwide
• up to 95%have no treatment approved for them
• 4.8 yrsaverage time from first symptom to a diagnosis

Those four numbers describe a gap, and models are already in it. Clinicians consult them on a case they will see once in a career. Parents consult them in year four of not knowing what their child has. Whatever a model says in that conversation carries more weight than it would anywhere else, because there is so little to weigh it against.

02 — WHAT I MEASUREThree questions, and none of them is accuracy

Accuracy on rare disease tasks is worth knowing and other people are measuring it. I am after something narrower: the behaviour of a model at the moment its knowledge runs out.

Calibration — does confidence still mean anything

A well-behaved model is one whose stated confidence tracks how often it is right. Say 90% and be right nine times in ten. My hypothesis is that this decouples here — confidence stays high while accuracy falls away underneath it, because fluency is produced by the same machinery whether or not the underlying facts exist.

Abstention — will it say that the evidence is not there

For most rare conditions the correct answer to a clinical question is that nobody knows yet. So a share of my items are built so that insufficient evidence is the only correct response. A model that produces a confident, well-written answer to one of them has failed the item, however good the paragraph is.

Grounding — what retrieval actually repairs

The standard answer is to give the model the literature. It clearly helps. What I have not seen measured is whether it fixes the failure or relocates it — either the model stops inventing, or it keeps being confidently wrong while now pointing at a real paper that does not say what it claims. The second is worse, because the citation makes it harder to catch.

03 — METHODWhere the right answers come from

There is an obvious objection to a designer building a medical benchmark, and it is the correct objection: who decides what the right answer is.

My answer is that I do not. Every item is drawn from a source where a qualified person has already made that call in public and defended it — published case reports with genetically confirmed diagnoses, variant classifications from expert panels, curated disease references. The gold label is theirs. My job is sampling, provenance and scoring, and those I can do rigorously. A clinical reviewer can delete an item, but never overrule its source.

04 — WHY MEWhere the question came from

I am a motion designer and director. I run a studio. I am not a clinician and I am not an academic, and I want that on the page rather than buried.

Through 2026 I have been running AI systems as working infrastructure inside that studio, not as demonstrations — brand systems, production websites, media pipelines, campaigns with deadlines and clients attached. What that year taught me is narrow but I think it transfers. The failure that costs you is almost never the model refusing. It is the model being fluent, certain and wrong in a place where nobody happens to look.

05 — STATUSWhat exists, and what I need

Nothing is published yet. The first item set is being assembled. No results are claimed anywhere on this page, and every figure above is a schematic of the question rather than a picture of an answer.

When there are findings, the items, their provenance, the scoring code and the results go out openly — including the runs that went badly. A benchmark whose failures are private is a marketing asset, not an instrument.

What I am looking for

• A rare disease clinician or genetics researcher willing to review items as a named collaborator. This is the piece I cannot supply myself and the piece the work most needs.
• Anyone working on calibration, abstention or evaluation who wants to argue with the design before it is built rather than after.
• Patient organisations who would want a say in which conditions the first item set covers.