How do you evaluate an AI model on clinical trial work?

Test the model on the decisions the work actually requires, such as consent, eligibility, safety reporting and deviations, scored with a rubric tied to ICH E6(R3), and compare it with practitioners who answered the same items.

Then state the context of use, the method and the limits in writing, because a score means something only for the task it was measured on, which is also how FDA's draft guidance frames the credibility of AI models (FDA draft guidance on AI, January 2025).

What a credible evaluation needs

  • A defined context of use: which decision, for whom, and what happens when the model is wrong.
  • Items written from real trial situations, with more than one plausible answer, not textbook recall.
  • A rubric that scores each response by its consequence for participants and data, tied to the section of ICH E6(R3) it tests.
  • A human baseline: practitioners in the relevant roles answering the same items under the same conditions.
  • Held-out items, so the evaluation is not memorized.
  • Published method, versions and corrections, so others can check the result.

Why multiple-choice recall is not enough

Models are strong at reciting regulations and weaker at applying them when the facts are messy: a participant who calls from a hospital on a weekend, an eligibility lab that arrives after dosing, a consent version that changed last week. Those are the decisions that matter at a site, and they need items with a scenario, a choice and a consequence.

What CRLN publishes

CRLN-AIE-001 is our published evaluation protocol. The AI scoreboard puts models and practitioners on the same published decisions, with its corrections in public, and the paired benchmark report compares people and AI on site decisions. Each version has a DOI and is licensed CC BY 4.0. Learner data is never used to train or tune any model.

Questions people also ask

What is a context of use for an AI model?

The specific role and scope of the model in answering a question of interest. FDA's draft guidance asks that a model's credibility be established for a particular context of use (FDA draft guidance on AI, January 2025).

Why compare AI with practitioners rather than with an answer key?

Because real decisions often have more than one defensible answer. A practitioner baseline shows whether the model is better, worse or different from the people who do the work.

Can a general benchmark like a medical exam tell you about trial operations?

Only a little. Clinical knowledge and trial conduct overlap, but GCP decisions about consent, delegation and reporting are a different skill and need their own items.

Does CRLN train models on learner answers?

No. Learner data is never used to train or tune a model, row-level learner records are never licensed, and nothing about a group of fewer than five people leaves the platform.

Primary sources

  • Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products, draft guidance (FDA): https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological
  • ICH E6(R3) Guideline for Good Clinical Practice, final version adopted 6 January 2025 (ICH): https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf

Last reviewed

2026-11-12