Over the past few years I have spent a great deal of time reading AI-generated scientific output: analyses, literature summaries, proposed mechanisms, suggested experiments. The most consistent lesson is that fluency is not evidence. Modern models write in the register of a competent scientist, which makes their mistakes harder to see, not easier.
That matters because life-science organizations are moving from experimenting with AI to depending on it. Once an output feeds a target decision, a compound prioritization or a regulatory document, the question is no longer “is this impressive?” but “is this right, and how would we know?”
Start from the decision, not the model
Validation is only meaningful relative to a use. The same model can be perfectly adequate for triaging literature and dangerously unreliable for proposing a mechanism of action. Before evaluating anything, write down the decision the output will inform and what it would cost to get it wrong. That determines how much evidence you need.
Five questions I ask of any scientific AI system
- What is the ground truth, and who defined it? Benchmarks built from textbook questions measure recall. Real research tasks involve ambiguous data, conflicting literature and judgment calls. If the evaluation set doesn’t look like your work, its scores don’t describe your risk.
- Where does it fail, specifically? An aggregate accuracy number hides the pattern. I want a catalogue of failure modes: misread assay context, conflated gene families, confident extrapolation beyond the data, citations that don’t say what they are claimed to say.
- Does it know when it doesn’t know? A system that flags uncertainty appropriately is far more useful than a slightly more accurate one that never hesitates.
- Can a scientist check the reasoning, not just the answer? Traceability to data and sources turns a black box into something a domain expert can audit in minutes rather than hours.
- What is the human-in-the-loop design? Decide in advance which outputs are used directly, which are reviewed, and which are never delegated. Write it down. Revisit it as the evidence accumulates.
Build the evaluation from real work
The most informative tests I have seen are drawn from an organization’s own past projects, where the answer is now known. Re-run a solved problem, compare the system’s reasoning with what actually happened, and have the scientists who did the work grade it. It is slower than running a public benchmark, and much more honest.
Validation is ongoing
Models change, data drifts, and workflows expand into territory nobody tested. A one-time assessment is a starting point. The organizations getting real value from scientific AI treat validation as a standing practice, with a small set of sentinel tasks re-checked whenever something changes.
None of this argues against using AI in research. It argues for using it the way we use any instrument: calibrated, understood, and with a clear sense of where its readings can be trusted.