Can AI pass the coordinator test?

We corrected ourselves in public, twice

12 September 2026, version 0.2. Version 0.2 reversed the conclusion of version 0.1. Version 0.1 said a frontier model and practitioners could not be told apart, but it had compared an unweighted model mean with a human mean weighted by how often each item was played. Popular items are easier items, so on the same data those two quantities differ by 0.093. Compared like with like, the conclusion reverses. Version 0.1 stays citable as a record, and none of its figures are current. (DOI 10.5281/zenodo.22728905; superseded version 0.1: DOI 10.5281/zenodo.22708752)

29 September 2026, version 0.2.1. Version 0.2.1, a correction only, said that the 57-scenario held-out set described in version 0.2 was not private, because its answers could be read without an account. That set is retired, and the one result that rested on it, the hard-tier comparison in section 5.5, is withdrawn as a measurement on unseen items. The 114-item results below are unaffected because those items were public by design. (DOI 10.5281/zenodo.23043069)

Panel 1. Same 114 decisions, models against practitioners

Mean rubric score from 1 to 4 for seven AI models and for practitioners on the same 114 decisions, CRLN-AIE-001 version 0.2, sorted by mean.

• claude-opus-5: Mean score (1 to 4) 3.846; Difference vs practitioners [95% CI] +0.335 [0.233, 0.434]; p value <0.0001; Well above; Meets the standard 98.9%; Compliance depended on option order (of 114) 1.
• gemini-3.6-flash: Mean score (1 to 4) 3.709; Difference vs practitioners [95% CI] +0.198 [0.092, 0.299]; p value 0.0001; Modestly above; Meets the standard 98.9%; Compliance depended on option order (of 114) 1.
• claude-sonnet-5: Mean score (1 to 4) 3.649; Difference vs practitioners [95% CI] +0.139 [0.035, 0.237]; p value 0.017; Modestly above; Meets the standard 98.2%; Compliance depended on option order (of 114) 3.
• gemini-3.7-flash: Mean score (1 to 4) 3.644; Difference vs practitioners [95% CI] +0.134 [0.027, 0.240]; p value 0.026; Modestly above; Meets the standard 98.9%; Compliance depended on option order (of 114) 1.
• gemini-3.8-flash: Mean score (1 to 4) 3.639; Difference vs practitioners [95% CI] +0.128 [0.023, 0.230]; p value 0.033; Modestly above; Meets the standard 98.6%; Compliance depended on option order (of 114) 2.
• claude-haiku-4-5-20251001: Mean score (1 to 4) 3.567; Difference vs practitioners [95% CI] +0.056 [-0.057, 0.161]; p value 0.353; Not distinguishable from practitioners; Meets the standard 96.1%; Compliance depended on option order (of 114) 6.
• gpt-4o-2024-08-06: Mean score (1 to 4) 3.496; Difference vs practitioners [95% CI] -0.014 [-0.135, 0.108]; p value 0.762; Not distinguishable from practitioners; Meets the standard 95.1%; Compliance depended on option order (of 114) 12.
• Practitioners (baseline): Mean score (1 to 4) 3.510.

Source: CRLN-AIE-001 version 0.2, section 4. Correction status for every row: version 0.2.1 retired the held-out set, and these 114-item rows are unaffected.

The last column counts decisions where the same model both met and failed the professional standard depending only on the order the options were shown in. It tracks capability: 1 for the strongest model, 12 for the weakest.

Ceiling. Order effects are only reliably detectable when a model meets the standard on roughly 60 to 95 percent of responses, and these models met it on 95.1 to 98.9 percent. A low count in the last column means not measurable here, not absent.

Panel 2. Do models say when they are unsure?

CRLN-PRE-002, pre-registered. This is a different measure on a different item set: 200 keyed items drawn with a registered seed, each shown in 3 option arrangements. Models replied with an option and a confidence from 1 (a guess) to 5 (certain). Accuracy here means choosing the authored best option, which is stricter than meeting the standard in Panel 1. Arm A, the original prompt, is shown.

• claude-opus-5: Accuracy vs the authored best option 85.8%; Answers at confidence 1 or 2 1 of 597.
• claude-sonnet-5: Accuracy vs the authored best option 77.0%; Answers at confidence 1 or 2 0 of 600.
• gemini-3.8-flash: Accuracy vs the authored best option 70.2%; Answers at confidence 1 or 2 0 of 600.
• gpt-4o-2024-08-06: Accuracy vs the authored best option 63.8%; Answers at confidence 1 or 2 1 of 600 (Addendum, not in the primary test).
• claude-haiku-4-5-20251001: Accuracy vs the authored best option 62.5%; Answers at confidence 1 or 2 0 of 600.

The registered test, on the Anthropic and Google models: labeling every point on the confidence scale moved low-confidence answers from 0.04% (1 of 2,397) to 0.17% (4 of 2,400), z = 1.34, p = 0.18. The null hypothesis holds.

Exploratory and not a matched comparison: practitioners on the platform used confidence 1 or 2 on 4.7% of 996 decisions from 49 learners, and 77 of their 118 below-standard decisions came at confidence 4 or 5. Neither side reliably flags its own mistakes, but people used the low end of the scale far more often than any model.

claude-haiku-4-5-20251001: a parser rule records its replies of a letter and a digit followed by a paragraph as non-answers (21 in Arm A), which lowers its measured accuracy.

gpt-4o-2024-08-06 was run later the same day under the identical registered command. It is reported in an addendum, section 10, and is not part of the primary test.

Source: CRLN-PRE-002 results, version 1.0.0, sections 1, 3, 5 and 10. DOI 10.5281/zenodo.23544672.

Method in plain language

• 114 clinical trial conduct decisions, each answered at least five times by practitioners.
• Every model saw every item 5 times, with the options re-randomized each time: 570 calls per model and 3,990 scored responses in all, at seed 7. A fixed-order control run separates order effects from retest noise.
• The practitioner baseline was frozen at 2026-09-11T00:00:00Z, before any model run, so it could not be re-cut after seeing results. It comes from 52 external learners, and the author's own accounts are excluded.
• Every response was retained with the vendor-reported model id and the option order as presented.
• Each option carries a rubric score from 1 to 4, and 3 or above meets the professional standard.

Limitations

• The design favors models. Authored multiple choice, curated vignettes, self-selected practitioners and a fixed answer key all push in the models' favor. Version 0.2 itself says models scoring above practitioners is what the research literature predicts for an instrument built this way.
• Public items. All 114 items are on public pages, so contamination cannot be excluded for any model.
• The practitioner baseline is concentrated. The heaviest ten of the 52 learners contribute about half the paired evidence, and dropping them moves the mean by 0.099, which is comparable to the effects reported.
• The comparison with people in Panel 2 is exploratory: a different item set, a different score and a different wording of the confidence scale.

What is not on the board, and why

• No OpenAI model newer than gpt-4o. The newer model we tried could not be run because the account was capped at 3 requests per minute for it.
• No flagship Gemini model. On run day every available flagship name was retired for new keys or carried a preview suffix, which would make a result impossible to reproduce.
• No held-out results. The version 0.2 held-out set is retired, and no held-out result for any model will be claimed until a new private set exists.

Next refresh

We plan to refresh this board with newer models under the same published protocol, with a new practitioner baseline frozen before any model run. A refresh appears here only after its results are published with a DOI. If a quarter passes with no new run, this page will say so.

Conflict of interest

CRLN operates this instrument and sells evaluations of AI models. That conflict is structural, which is why the limitations sit next to the results.

We would rather be refuted than cited. If you find an error, tell us. A wrong figure is corrected in a new version, and the old one stays citable.

For AI teams

If your model will be used near clinical trial work, we can evaluate it under the same published protocol, against a practitioner baseline frozen before any model is run, with every item shown several times in shuffled order. Results are delivered in aggregate, and a client's model appears on this board only with the client's written consent.