AI Screened 200,000 Medical Papers for $856. It Missed Almost Nothing

Somewhere in a database of 201,064 studies sits the answer to a question no one has time to ask. Does raising the minimum wage lower the suicide rate? Does expanding health insurance? For years the honest answer from researchers was: we cannot check, because reading a fifth of a million abstracts by hand would take a team the better part of a year and cost as much as a house.

A group at Johns Hopkins just did the reading part for $855.91.

Their tool, an artificial intelligence agent called ScreenAgent, went through every one of those records and threw out 99.4% of them, leaving a manageable pile for humans to finish. It caught 43 of the 44 studies that actually belonged in the review.

What happened#

On 12 August 2026, researchers from the Center for Suicide Prevention at the Johns Hopkins Bloomberg School of Public Health posted a preprint on medRxiv describing ScreenAgent, a large language model agent built to do the first and most tedious step of a systematic review: reading titles and abstracts and deciding which studies are worth a closer look. This is a preprint and has not yet been peer reviewed.

The team tested it three ways. First, on their own live meta-analysis of suicide-prevention research, where the agent screened the full haul of 201,064 records. Then on two systematic reviews that had already been published, so the "right" answers were known in advance.

The headline numbers are hard to argue with. On the internal review, ScreenAgent recovered 43 of the 44 studies that human experts had judged eligible, a sensitivity of 97.7%. It agreed with the humans' final consensus slightly more often than the humans agreed with each other (a Cohen's kappa of 0.75 for the agent against 0.64 between human reviewer pairs). And it did the whole 201,064-record sweep for $855.91, which works out to about $4.26 per thousand papers, or under half a penny each. A second, higher-effort check by a more powerful model added another $57.70. Set that against the going rate for a systematic review, which one widely cited analysis put at roughly $141,000 and eleven months of work, and the shift in economics is obvious.

Why finding papers is the real bottleneck#

A systematic review is the closest thing medicine has to a definitive answer. Rather than trusting any single study, researchers gather every relevant study on a question, weigh them together, and often pool the numbers into a meta-analysis. These sit at the top of the evidence pyramid. They shape clinical guidelines, insurance decisions, and government policy.

The catch is the front door. To be thorough, you cast a wide net in databases like PubMed, and that net comes back full. A broad search can return tens or hundreds of thousands of records, the overwhelming majority of them irrelevant. Two humans then read each title and abstract independently and decide, one by one, keep or discard. This stage is called screening, and it is where reviews go to die. It is slow, mind-numbing, and expensive.

So reviewers cheat, sensibly. They write narrow searches to keep the pile small enough to read. That keeps the project doable, but it silently decides what the review can ever find. As the authors point out, a tightly worded search for "suicide prevention" will sail straight past studies on firearm laws, alcohol pricing, housing support, or insurance expansion, even though those policies can move suicide rates. The evidence exists. The search was just never allowed to be wide enough to reach it.

That is the constraint ScreenAgent is really attacking. Not reading speed for its own sake, but the trade-off between how broadly you can search and how much you can afford to read.

Why this matters beyond suicide research#

Every field that leans on evidence synthesis faces the same wall: cardiology, oncology, public health, drug safety, even areas of biology and economics. If broad screening becomes cheap and reliable, reviewers can stop pruning their questions to fit their labour budget. They can ask the bigger, messier questions where the useful evidence is scattered across disciplines and buried under unhelpful keywords.

There is a knock-on effect for how fast evidence reaches the people who use it. A guideline committee that would have waited a year for a review might get a defensible answer in weeks. During a fast-moving health emergency, that gap between "the evidence exists" and "we have read the evidence" is not academic. It is the difference between a policy grounded in what is known and one made in the dark.

There is also a quieter contribution that researchers may end up valuing more than the cost savings. The team built a way to estimate, in advance, how good the screening is likely to be on a given review, complete with confidence intervals. Screening has always been a black box you had to trust. This turns it into something you can measure and report, which is exactly what a nervous journal editor or guideline panel will want before they rely on an algorithm's judgement.

What the study does not settle#

Start with that single missed study. In a suicide-prevention review, one overlooked paper could mean an effective intervention never makes it into the evidence base. The one ScreenAgent missed was a genuinely borderline case that the human reviewers themselves had argued over. That is reassuring, but it does not make the miss disappear. This is why the team designed the system to assist rather than replace: the agent does a first pass, a second stronger model double-checks anything it flags, and humans still make every final call.

Then there is generalisation. Every corpus the team tested was about suicide-related outcomes. The framework is not built for that topic specifically, but its performance everywhere else is, for now, a promise rather than a result. The external tests hint at the hidden work involved. When the researchers pointed a minimal, generic prompt at two unfamiliar reviews, sensitivity sagged to 80.6% and 89.5%. What rescued it was not a smarter model but careful tuning of how each review's eligibility rules were written into the prompt, after which sensitivity climbed back to 95.9% and 97.4%. In other words, you do not just switch this on. Someone who knows what they are doing has to configure and validate it for each new question.

The results also ride on specific commercial models, and models change. The best performers were the largest ones running at medium reasoning effort. Smaller, cheaper models fell away badly, with one dropping to 70.5% sensitivity. Since these systems are updated and retired on the vendors' schedule, any screening pipeline built on them will need re-checking every time the underlying model shifts. Automation, in this sense, does not end the validation work. It moves it.

How this compares with what came before#

AI-assisted screening is not new. Tools that rank or filter abstracts with machine learning have been around for years, and the field has been inching toward high-performance automated screening for a while. What is different here is the combination: near-complete capture of eligible studies, reliability at the level of trained humans, an honest way to estimate performance ahead of time, and a cost low enough to make the old "narrow your search" compromise unnecessary. Earlier tools mostly made screening faster. This one aims to make broad screening trustworthy.

The wider context is a useful corrective to hype. Large language models routinely ace medical licensing exams, yet an Oxford study published in Nature Medicine this year found that when ordinary people actually used LLMs to work out what was wrong with them, they did no better than people using a normal web search. Benchmark brilliance and real-world usefulness are not the same thing. ScreenAgent looks like a case where the task is well matched to what these models are genuinely good at: reading a lot of text quickly and applying a consistent rule, under human supervision, with a clear right answer to check against.

That framing lines up with where the expert consensus has settled. The Cochrane Rapid Reviews Methods Group, which sets much of the standard for this kind of work, backs using automation to speed reviews while keeping human experts as the final decision-makers. ScreenAgent is built to fit that mould rather than break it.

Key takeaways#

1. A Johns Hopkins AI agent, ScreenAgent, screened 201,064 studies for a live suicide-prevention meta-analysis, removing 99.4% of records from human review at a total cost of $855.91.
2. It found 43 of 44 eligible studies (97.7% sensitivity) and agreed with the human consensus slightly more often than the human reviewers agreed with each other.
3. The real prize is not speed but breadth: cheap, reliable screening lets researchers run wide searches that catch cross-cutting evidence which narrow searches systematically miss.
4. It is not autonomous. A second model double-checks flagged papers, humans make the final decisions, and each new review needs its own tuning and validation.
5. This is a preprint, not peer-reviewed, and every test so far was on suicide-related literature. Its performance in other fields, and its dependence on specific models, both remain open questions.

Frequently asked questions#

Is ScreenAgent replacing human reviewers? No. It handles the first-pass sort, discarding clearly irrelevant papers. A stronger model rechecks what it keeps, and humans make every final inclusion decision. The authors describe it as a prescreening assistant, not a replacement.

How can screening 200,000 papers cost only $856? The agent reads each title and abstract and returns a simple keep-or-discard decision, which is a small task per record. At scale, and using batch pricing, that came to roughly $4.26 per thousand records, or under half a penny each.

Does 97.7% sensitivity mean it is safe to use now? It means it caught 43 of 44 eligible studies in one internal review, which is strong but not perfect. The one miss was a borderline case. For high-stakes reviews the missed-study risk is exactly why humans stay in the loop.

Would this work for my field, not just suicide research? Possibly, but it has not been shown yet. Every test was on suicide-related literature. The external results also showed that moving to a new review required careful, expert prompt tuning before performance recovered.

Which AI models did it use? The team tested several. The largest models at medium reasoning effort performed best and tied at the top; smaller, cheaper models did markedly worse. Because vendors update and retire models, any real pipeline would need re-validation whenever the model changes.

Is this peer-reviewed? No. It is a preprint posted on medRxiv on 12 August 2026. Preprints share findings early, before independent review, so the results should be treated as promising rather than settled.

What is the single biggest limitation? Generalisation. The method is not built for suicide research specifically, but until it is tested across other clinical and policy domains, its wider performance is an expectation, not a proven fact.

Glossary#

Systematic review: a structured summary of all the studies relevant to a specific question, designed to reduce bias by not cherry-picking evidence.

Meta-analysis: a statistical method that pools the numerical results of several studies to produce a single, more precise estimate.

Screening: the stage where reviewers read titles and abstracts and decide which studies to keep for full review. The main bottleneck this work targets.

Sensitivity: the share of truly eligible studies a screening method correctly keeps. High sensitivity means few relevant papers are missed.

Specificity: the share of irrelevant studies correctly discarded. High specificity means less wasted human effort on papers that do not belong.

Cohen's kappa: a score for how well two reviewers agree beyond what chance alone would produce. Higher is better. Here it compared the agent against humans, and humans against each other.

Large language model (LLM): an AI system trained on huge amounts of text that can read and generate language, and follow instructions such as "keep this paper or discard it."

LLM agent: an LLM given a defined job and a workflow to run through, rather than a single chat reply. ScreenAgent is one built specifically to sort research papers.

References#

1. Dobin D, Witmer AM, Sweeney FG, Ryan T, Cimino A, Haroz EE, Nestadt PS, Wilcox HC. "Making Broad Evidence Synthesis Feasible: An LLM Screening Agent for Meta-Analyses Applied To Suicide Prevention." medRxiv, 12 August 2026. Preprint (not peer reviewed). https://doi.org/10.64898/2026.08.12.26360335
2. Michelson M, Reuter K. "The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials." Contemporary Clinical Trials Communications, 2019. https://www.sciencedirect.com/science/article/pii/S2451865419302054
3. Garritty C, et al. Cochrane Rapid Reviews Methods Group guidance on the use of automation in rapid reviews. Cochrane Evidence Synthesis and Methods, 2024. https://doi.org/10.1002/cesm.70063
4. Prior work on high-performance automated abstract screening. Journal of the American Medical Informatics Association (JAMIA), 2025. https://doi.org/10.1093/jamia/ocaf050
5. University of Oxford. "New study warns of risks in AI chatbots giving medical advice" (on "Clinical knowledge in LLMs does not translate to human interactions," Nature Medicine, 2026). https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice
6. medRxiv abstract page for ScreenAgent. https://www.medrxiv.org/content/10.64898/2026.08.12.26360335v1

A note on the subject matter: this article discusses research into suicide prevention. It covers methods for reviewing scientific literature and contains no information about self-harm. If you are affected by these issues, please consider reaching out to a trusted person or a local support service.