Hospitals Are Quietly Handing Triage to AI. What Does the Evidence Show?

The nurse, the queue and the algorithm in the corner#

Walk into a large emergency department on a Friday night and the first clinical decision about you is made within minutes, usually by a triage nurse with a blood pressure cuff, a short conversation and a scoring system. That decision sets how long you wait. It also turns out to be wrong surprisingly often. A study of more than 5.3 million adult visits across 21 Kaiser Permanente emergency departments in California found that 32.2% of patients were mistriaged under the most widely used US system (Sax et al., 2023).

That number explains why hospitals have been so keen on AI triage. A tool that never tires, reads the whole record and gives the same answer at 4am as at 4pm sounds like an obvious upgrade. Some of these tools now run quietly inside electronic health records, nudging nurses towards a higher or lower priority, flagging possible sepsis, or pinging a stroke team before a radiologist has opened the scan.

The awkward part is the evidence. A 2026 systematic review of AI and machine learning triage models in emergency departments found 14 eligible studies, every one of them retrospective, and no prospective or randomised trials at all (Mohamed et al., 2026). So the answer to "does it work?" is more interesting than yes or no. It depends on which tool, which task, and what you mean by "work".

Background: what triage is, and how you score a machine#

The job triage is supposed to do#

Triage is sorting. Its purpose is to make sure the sickest patients are seen first when there are more patients than clinicians. In the US, most emergency departments use the Emergency Severity Index (ESI), a five-level scale running from level 1 (needs immediate life-saving intervention) to level 5 (could safely wait). The UK and much of Europe use similar five-level systems, such as the Manchester Triage System.

Two errors matter. Undertriage means a patient is rated as less urgent than they really are, which is the dangerous one because people wait while deteriorating. Overtriage means a patient is rated as more urgent than necessary, which clogs the fast lane and slows care for everyone else. In the Kaiser Permanente data, 3.3% of visits were undertriaged and 28.9% overtriaged, and ESI correctly placed only 65.9% of the patients who went on to need a life-stabilising intervention into the top two levels (Sax et al., 2023).

Most patients land in the middle. In a Johns Hopkins study, more than 65% of emergency visits were assigned ESI level 3, a large grey zone where people with a sprained ankle and people with an evolving heart attack can end up in the same queue (Levin et al., 2018).

The three flavours of AI triage#

"AI triage" covers several quite different technologies, and it helps to keep them apart.

The first is classic machine learning: statistical models such as random forests or gradient-boosted trees that learn patterns from thousands of past visits. They take vital signs, age, the chief complaint and medical history, then estimate the chance of outcomes like intensive care admission. The second is task-specific clinical decision support, such as software that reads a CT angiogram for a blocked brain artery or scans the record for early signs of sepsis. The third, and newest, is the large language model (LLM), the same family of technology behind ChatGPT, which reads free-text notes and produces an acuity rating or a diagnosis in plain language.

How to read the numbers#

You will see a statistic called AUC (area under the receiver operating characteristic curve) in nearly every paper. It measures how well a model ranks patients: 0.5 is a coin toss, 1.0 is perfect, and anything above 0.8 is usually called good. AUC does not tell you whether the model's predicted probabilities are accurate (that is "calibration"), nor whether using the model changes what happens to patients. The 2026 review found calibration was reported in only three of 14 studies and external validation (testing in a different hospital from the one that built the model) in only five (Mohamed et al., 2026). Keep that gap in mind. It comes up again and again.

What retrospective studies promise#

The early results were genuinely encouraging. In 2018, a Johns Hopkins team tested an electronic triage system, built on a random forest model, against ESI across 172,726 visits at two emergency departments. Its AUCs ranged from 0.73 to 0.92, and it picked out more than 10% of ESI level 3 patients who should have been moved up. Those up-triaged patients had a 6.2% chance of needing critical care or an emergency procedure, against 1.7% for level 3 as a whole (Levin et al., 2018). That tool later became a commercial product, which is one route by which research models end up in real hospitals.

Reviews since then tell a consistent story. An integrative review of 26 studies reported that machine learning triage models often achieved AUCs above 0.80 for outcomes such as hospital admission and transfer to intensive care, and that adding free-text notes through natural language processing improved accuracy (El Arab & Al Moosa, 2025). A separate scoping review of 19 studies concluded that machine learning models "consistently outperformed" traditional triage systems on prediction, while warning that many tools still lack the reliability needed for large-scale clinical use (Souza et al., 2026).

Here is the catch. Predicting who will be admitted, using data that has already been tidied up after the event, is easier than making a live decision at the front door. Retrospective data also bakes in the habits of the hospital where it was collected. A model can learn that patients who arrive by ambulance get admitted without learning anything about why. Strong AUCs are a reason to run a trial, not a substitute for one.

When deployed algorithms meet reality#

The best-known cautionary tale is not strictly a front-door triage tool, but it shows what happens when a prediction model spreads faster than its evaluation. The Epic Sepsis Model, a proprietary tool built into a widely used electronic health record, was running at hundreds of US hospitals when University of Michigan researchers tested it independently on 38,455 hospital stays. Its AUC was 0.63. It missed 67% of patients who developed sepsis, while generating alerts for 18% of all hospitalised patients, a recipe for alert fatigue (Wong et al., 2021).

Contrast that with TREWS, a sepsis early warning system developed at Johns Hopkins and studied prospectively across five hospitals. Among 6,877 patients with sepsis flagged before antibiotics began, those whose alert was confirmed by a clinician within three hours had an adjusted 3.3 percentage-point absolute reduction in in-hospital mortality, an 18.7% relative reduction, compared with patients whose alert was not confirmed in that window (Adams et al., 2022). That is a meaningful result. It is also observational: the comparison is between clinicians who did and did not respond quickly, and those groups may differ in ways the adjustment cannot fully capture.

Image-based triage has the cleanest evidence so far. In a cluster randomised stepped-wedge trial at four Houston stroke centres, software that automatically detected large vessel occlusions on CT angiograms and sent alerts to clinicians' phones cut door-to-groin time (arrival to the start of clot removal) by 11.2 minutes. It did not, however, show a significant difference in patients' functional independence at 90 days (Martinez-Gutierrez et al., 2023). Faster is good for stroke, where every minute counts, but the trial was not large enough to prove better outcomes.

Ambulance dispatch is getting trials too. The Swedish MADLAD randomised trial gave dispatchers machine learning risk scores when there were more low-priority patients than ambulances. With the tool, the first free ambulance went to the sickest patient (judged by a later early warning score) 68.3% of the time, compared with 62.5% under usual practice, an odds ratio of 1.28 that only just reached statistical significance. The authors note the trial was underpowered (Spangler et al., 2026).

The table below lines up the main studies by the strength of their design.

Large language models arrive at the front door#

The newest entrants are the chatbots, and they are arriving with impressive headlines. In April 2026, researchers from Harvard Medical School and Beth Israel Deaconess Medical Center reported in Science that a reasoning LLM outperformed physician baselines across a series of clinical reasoning experiments, including a real-world comparison of AI and human second opinions on randomly selected emergency patients (Brodeur et al., 2026). According to press coverage of the study, OpenAI's o1 model matched the exact or near-exact diagnosis for 67% of 76 emergency patients at the triage stage, against 50 to 55% for pairs of emergency physicians working from the same records (Resultsense, 2026). The authors themselves concluded that this motivates "the urgent need for prospective trials", not immediate deployment (Brodeur et al., 2026).

Two things are worth noticing. First, that study tested diagnosis from text records, which is related to triage but not the same job. Second, studies that ask LLMs to do the actual triage task produce more sobering numbers. A University of California, San Francisco team found GPT-4 correctly picked the sicker of two patients in 89% of 10,000 pairs, comparable to a physician reviewer (Williams et al., 2024). Choosing between two patients whose ESI scores differ is easier than assigning one of five levels, though.

When a Greek emergency department tested seven LLMs on 39,375 real patients against physician-assigned ESI levels, the best models (DeepSeek and Claude Sonnet 4) agreed with physicians only 61.7% of the time, with a weighted kappa of about 0.47. Kappa is a measure of agreement that corrects for chance, and 0.47 counts as moderate. GPT-5 Instant performed poorly, and no model reached strong agreement. The models also tended towards both over- and undertriage (Nedos et al., 2026). A 2026 scoping review of 27 studies summed up the LLM evidence as "preliminary", citing concerns about undertriage, inconsistency and hallucination (the confident generation of false information) (Hikmat et al., 2026).

The chart below puts three of these findings side by side. Each bar is 100% of its own study population, so read them as separate snapshots rather than a league table.

Sources: Sax et al., 2023; Wong et al., 2021; Nedos et al., 2026. Note that in the LLM study the reference standard is the physician's own ESI score, which is itself imperfect.

That last caveat deserves a moment. If human triage is wrong a third of the time, then grading an AI by how often it agrees with humans measures imitation, not accuracy. Some of the "disagreements" may be the model being right. We cannot tell from that design, which is exactly why outcome-based trials matter.

Bias, automation bias and the human in the loop#

AI triage can make old inequities faster and harder to see. The Kaiser Permanente study already found that Black patients had an 18.5% higher relative risk of undertriage than White patients under human ESI scoring (Sax et al., 2023). A model trained on those decisions can learn the same pattern. The classic warning comes from a 2019 Science paper on a commercial algorithm used to steer extra care to high-risk patients: because it predicted healthcare costs rather than illness, and less money was spent on Black patients, it judged them healthier than equally sick White patients. Fixing the target would have raised the share of Black patients flagged for extra help from 17.7% to 46.5% (Obermeyer et al., 2019). Triage models that learn from "was admitted" or "went to intensive care" carry a similar risk, because those outcomes reflect access and habit as well as need.

The second problem sits on the human side. Hospitals usually describe AI triage as advisory, with a nurse or doctor making the final call. That only works if people push back when the machine is wrong. A controlled experiment with 27 radiologists reading mammograms showed how hard that is. When a purported AI suggested the wrong category, accuracy fell from about 80% to 20% among inexperienced readers, and from 82% to 46% even among very experienced ones (Dratsch et al., 2023). This tendency to defer to automated suggestions is called automation bias. A 2026 review of the state of clinical AI flagged automation bias and clinician deskilling as emerging risks, and noted that human and AI collaboration "remains suboptimal" (Worth et al., 2026).

There is also the patient-facing end of triage: online symptom checkers that tell people whether to call an ambulance or stay home. A systematic review led by Imperial College London found triage accuracy ranging from 48.8% to 90.1% across tools, and warned that reliance on them "could pose significant patient safety hazards" (Wallace et al., 2022).

Who checks the checker? Regulation and the evidence gap#

The word "quietly" in this article's headline is not paranoia. Many triage aids sit in a regulatory grey zone. In the US, software that supports rather than replaces a clinician's judgement can fall outside medical device rules if the clinician can independently review the basis for its recommendation. In January 2026, the US Food and Drug Administration (FDA) revised its clinical decision support guidance. The update moved the limit on "time-critical" decision-making into the criterion about whether clinicians can independently review a recommendation, and extended enforcement discretion to tools that give a single recommendation when only one is clinically appropriate (FDA, 2026; Covington, 2026). Legal analysts note that the FDA did not define what counts as time-critical, which matters a great deal for emergency triage (Covington, 2026).

Cleared devices are not necessarily well studied either. The 2026 state-of-clinical-AI review reported that fewer than 5% of FDA-cleared AI and machine learning devices have undergone peer-reviewed evaluation, and that prospective randomised trials remain scarce (Worth et al., 2026). The same review concluded that the most credible progress comes from narrowly defined tasks with clear endpoints (Worth et al., 2026). That matches the triage evidence neatly: stroke detection, a single, well-defined image question, has a randomised trial behind it, while general-purpose acuity scoring mostly does not.

So, what would good look like? The reviews converge on a short list: external validation at several sites, reported calibration, fairness checks across patient groups, monitoring after deployment, and randomised or stepped-wedge trials that measure what patients care about, such as deaths, harm and waiting time (Mohamed et al., 2026; Hikmat et al., 2026). If you are a hospital buyer, a founder or a regulator, those are the questions to ask before the tool goes live, not after.

The AI triage is neither a fad nor ready to run unsupervised. The narrow tools are earning their place. The broad ones, especially LLMs, are promising students who have not yet sat the exam that counts.

Frequently asked questions#

Is AI already making triage decisions in hospitals? In some places, AI tools already influence triage, though usually as advice rather than a final decision. Commercial models grew out of research such as the Johns Hopkins e-triage system (Levin et al., 2018), and sepsis prediction tools have run at hundreds of US hospitals (Wong et al., 2021).

Is AI triage more accurate than a triage nurse? In retrospective studies, machine learning models often predict outcomes better than standard scales such as ESI (El Arab & Al Moosa, 2025). In live use the evidence is thin: a 2026 review found no randomised trials of general emergency department triage models (Mohamed et al., 2026).

What is the Emergency Severity Index? ESI is a five-level triage scale used in most US emergency departments, from level 1 (immediate) to level 5 (least urgent). In one large study it misclassified 32.2% of adult visits (Sax et al., 2023).

Can ChatGPT-style models triage patients safely? Not on their own yet. In a study of 39,375 patients, the best of seven LLMs agreed with physicians' ESI scores 61.7% of the time, and the authors recommended supervised use only (Nedos et al., 2026).

Has any AI triage tool improved patient outcomes? A prospective study of the TREWS sepsis alert linked prompt clinician confirmation to an 18.7% relative reduction in in-hospital mortality (Adams et al., 2022). A randomised stroke trial showed faster treatment but no proven difference in disability (Martinez-Gutierrez et al., 2023).

Can AI triage be biased? Yes. Human triage already shows racial disparities in undertriage (Sax et al., 2023), and algorithms trained on cost or admission data can inherit them (Obermeyer et al., 2019).

What is automation bias? Automation bias is the tendency to follow an automated suggestion even when it is wrong. In one experiment, even very experienced radiologists' accuracy fell from 82% to 46% when given incorrect AI advice (Dratsch et al., 2023).

Are AI triage tools regulated? Partly. In the US, the FDA's January 2026 clinical decision support guidance sets out when software counts as a medical device, but fewer than 5% of cleared AI devices have peer-reviewed evaluations (FDA, 2026; Worth et al., 2026).

References#

1. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nature Medicine. 2022;28(7):1455-1460. https://doi.org/10.1038/s41591-022-01894-0
2. Brodeur PG, Buckley TA, Kanjee Z, et al. Performance of a large language model on the reasoning tasks of a physician. Science. 2026;392(6797):524-527. https://doi.org/10.1126/science.adz4433
3. Covington & Burling. 5 key takeaways from FDA's revised clinical decision support (CDS) software guidance. January 2026. Link
4. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology. 2023;307(4):e222176. https://doi.org/10.1148/radiol.222176
5. El Arab RA, Al Moosa OA. The role of AI in emergency department triage: an integrative systematic review. Intensive and Critical Care Nursing. 2025;89:104058. https://doi.org/10.1016/j.iccn.2025.104058
6. US Food and Drug Administration. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. January 2026. https://www.fda.gov/media/109618/download
7. Hikmat R, Supriyadi Y, Noor F, et al. Artificial intelligence-assisted triage in emergency departments: a scoping review of clinical applications, outcomes, and implementation challenges. Clinical and Experimental Emergency Medicine. 2026. https://doi.org/10.15441/ceem.26.283
8. Levin S, Toerper M, Hamrock E, et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the Emergency Severity Index. Annals of Emergency Medicine. 2018;71(5):565-574.e2. https://doi.org/10.1016/j.annemergmed.2017.08.005
9. Martinez-Gutierrez JC, Kim Y, Salazar-Marioni S, et al. Automated large vessel occlusion detection software and thrombectomy treatment times: a cluster randomized clinical trial. JAMA Neurology. 2023;80(11):1182-1190. https://doi.org/10.1001/jamaneurol.2023.3206
10. Mohamed N, Mohammed Ibrahim AA, Shahid N, et al. Artificial intelligence and machine learning-based triage systems in emergency departments: a systematic review of predictive performance and clinical outcomes. Cureus. 2026;18(6):e111230. https://doi.org/10.7759/cureus.111230
11. Nedos I, Zagalioti SC, Kofos C, et al. Is artificial intelligence ready for emergency department triage? A retrospective evaluation of multiple large language models in 39,375 patients at a university emergency department. Journal of Clinical Medicine. 2026;15(4):1512. https://doi.org/10.3390/jcm15041512
12. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. https://doi.org/10.1126/science.aax2342
13. Resultsense. Harvard: AI beats ER doctors on text-based triage. 1 May 2026 (science journalism, reporting on reference 2). Link
14. Sax DR, Warton EM, Mark DG, et al. Evaluation of the Emergency Severity Index in US emergency departments for the rate of mistriage. JAMA Network Open. 2023;6(3):e233404. https://doi.org/10.1001/jamanetworkopen.2023.3404
15. Souza LL, de Oliveira YCN, Campos da Costa LC, et al. Artificial intelligence in emergency department triage: a scoping review. PLoS One. 2026;21(6):e0352338. https://doi.org/10.1371/journal.pone.0352338
16. Spangler DN, Morelli S, Smekal D, Edmark L, Blomberg H. Machine learning assisted differentiation of low acuity patients at dispatch: the MADLAD randomized controlled trial. PLoS Medicine. 2026;23(3):e1004770. https://doi.org/10.1371/journal.pmed.1004770
17. Wallace W, Chan C, Chidambaram S, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. npj Digital Medicine. 2022;5(1):118. https://doi.org/10.1038/s41746-022-00667-w
18. Williams CYK, Zack T, Miao BY, et al. Use of a large language model to assess clinical acuity of adults in the emergency department. JAMA Network Open. 2024;7(5):e248895. https://doi.org/10.1001/jamanetworkopen.2024.8895
19. Worth JE, Perez A, Wu DJH, et al. State of clinical AI in 2026. BMJ Digital Health & AI. 2026;2(1):e000113. https://doi.org/10.1136/bmjdh-2026-000113