How trial-operations AI decides, compared with Practice Lab learners

Models run October 12, 2026 (UTC). Human baseline frozen October 11, 2026, before any model call.

We give AI models from Anthropic, OpenAI and Google the same clinical research site decisions that clinical research staff and people training for those roles answer in the CRLN Practice Lab, one decision at a time, and compare the choices. This page reports a small public set of those decisions. Results are dated and tied to the exact model version that answered.

Read this first

  • These decisions come from CRLN's public practice catalog. Their text is readable in the free Practice Lab and their scoring reached browsers until October 5, 2026, so a model may have seen them. A high score here is not evidence about decisions outside this public set.
  • Every model that completed this edition chose a response our rubric scores as competent at least 97% of the time. This set cannot tell these models apart on competence. That is a limit of the items, and it is why CRLN is writing a new set of harder items that never enter the app.
  • Some items were drafted with help from Claude, an Anthropic model. Item families known to be drafted with Claude are excluded, but the rest do not record per-item authorship, so this set may favor Claude models.

Results, edition 2026-10

Each figure is a pooled rate over items with a 95% interval (bootstrap over items). Models answered every item 5 times with the options reshuffled each time.

  • Practice Lab learners (pooled): 90.1% competent decisions, 45 items with 5 or more learners
  • claude-fable-5-1 (Anthropic): 100.0% competent decisions
  • claude-opus-5-5 (Anthropic): 100.0% competent decisions
  • claude-sonnet-5-5 (Anthropic): 100.0% competent decisions
  • claude-haiku-5-5 (Anthropic): 100.0% competent decisions
  • gpt-6-astra (OpenAI): 100.0% competent decisions
  • gpt-4o (OpenAI): 97.3% competent decisions
  • gemini-3.8-flash (Google): 99.6% competent decisions

How to read these results

  • Competent decisions: the share of answers our rubric scores as meeting the standard or better. Our rubric is CRLN's own and has not been independently reviewed item by item.
  • Agrees with practicing reviewers: on items where a strict majority of three or more independent practicing professionals chose the same competent response, how often the responder chose that response. Only 4 public items meet this rule so far, the reviewers worked in one role, and they rated while our public site still showed the scored response. Treat this column as a first look, not a finding.
  • Many items have two responses the rubric scores as competent. When a model disagreed with the reviewers in this edition, it chose the other competent response, not one below the standard.
  • Stability: when a model changed its answer across reshuffles, it almost always moved between two competent responses. No model moved across the standard on more than 2 of 64 items; the full stability and position figures are in the results file.

The human baseline

Practicing and aspiring clinical research staff answering the same decisions in the CRLN Practice Lab, first attempt only, with test and staff accounts excluded. Human figures are pooled over items that each have 5 or more learners, and a human figure is shown only when it pools at least 3 such items. No individual's answers, and no single item's human result, leave CRLN. Learners see a scenario's steps in order with feedback, while a model sees each step alone, which favors learners on later steps.

Governance

  • CRLN sells AI evaluations. No AI developer paid to be included, and buying an evaluation from CRLN neither adds nor removes a row here. Paid evaluations use a separate private item set, never these public items.
  • Vendors may submit: ask CRLN to run a named model or agent under this protocol. Rows are labeled by who ran them.
  • CRLN publishes its methods. A vendor cannot choose the items, choose which measures appear, or remove a row.
  • Corrections: a reported error is checked within 10 business days. A confirmed error is corrected with a dated note at the top of this page, and the superseded figure stays visible in the note.
  • Some catalog items were drafted with help from AI models. Item families drafted with Claude are excluded, but other items do not record per-item authorship, so an advantage for Claude models from self-authorship cannot be ruled out.

What this page does not show

Performance on decisions a model cannot have seen, performance of a specific product or agent, other roles, and other languages. Those come from CRLN's private decision sets, which stay private and do not appear on this page.

Common questions

How do AI models compare with people on clinical trial site decisions?

In CRLN's benchmark, edition 2026-10, 7 models from Anthropic, OpenAI and Google each chose a response our rubric scores as competent between 97.3% and 100.0% of the time, against 90.1% for Practice Lab learners. These decisions come from CRLN's public practice catalog, so a model may have seen them, and learners see a scenario's steps in order with feedback while a model sees each step alone. The set cannot tell these models apart on competence, and it is not evidence about decisions outside this public set.

Why is a high score on a fixed set of decisions not enough to evaluate trial AI?

When models sit near the ceiling, the share of competent decisions stops separating them. This benchmark therefore reports three measures that still differ between models: agreement with practicing reviewers on the 4 items where a strict majority of three or more reviewers chose the same competent response, stated confidence (how often a responder was below standard when it said it was certain), and stability, meaning whether the same option is chosen when the options are reshuffled. The reviewer agreement column is a first look, not a finding.

Who paid for this benchmark?

No AI developer paid to be included. CRLN sells AI evaluations, and buying an evaluation from CRLN neither adds nor removes a row here. Paid evaluations use a separate private item set, never these public items. A vendor cannot choose the items, choose which measures appear, or remove a row.

What does this benchmark not show?

Performance on decisions a model cannot have seen, performance of a specific product or agent, other roles, and other languages. Those come from CRLN's private decision sets, which stay private and do not appear on this page.

How to cite

CRLN (2026). CRLN-AIE-PUB, edition 2026-10: how trial-operations AI decides, compared with Practice Lab learners. Models run October 12, 2026. https://crln-learn.com/findings/ai-benchmark