# An external validation of the Epic Sepsis Model at Michigan Medicine found an AUC of 0.63, missing 67% of sepsis patients while alerting on 18% of all hospitalised patients

Page: https://flyvolo.ai/en/changes/ev-20210621-registered-nurse-4
Date: 2021-06-21 · Stage: Constraint
Can this record move a task judgement? yes
Occupation: https://flyvolo.ai/en/careers/registered-nurse

## Scope

27,697 patients over 38,455 hospitalisations at one US academic health system, 6 December 2018 to 20 October 2019; sepsis in 6.6% of hospitalisations. At the implemented threshold the model found 183 of 2,552 sepsis patients (7%) that clinical care had missed. The vendor's own reported AUC was 0.76-0.83. One widely deployed proprietary model at one site — not every early-warning tool, and Epic revised the model afterwards.

## Source

- JAMA Internal Medicine — Wong et al. 181(8) (open-access copy, PMC8218233) — https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/ (primary source)

## What this means

The model was live in a real hospital, and its own numbers show what "watching the patient" costs to automate: at the deployed threshold it alerted on nearly one in five admissions while missing two-thirds of the sepsis cases. The scarce thing is not detection but calibration — a bedside judgement that fires rarely enough to still be believed.

## What it does not show yet

It does not show that predictive monitoring is a dead end. The study is one proprietary model at one site over eleven months, and Epic rebuilt it afterwards. It also does not show that the nurse keeps the hours: an alert nobody trusts still has to be assessed and documented, so a badly calibrated tool can add work while removing none.

## How to verify it yourself

If you work in a unit with an early-warning score, ask two questions with numbers attached: how many alerts fire per shift, and how many of those changed what anyone did. Any vendor AUC quoted without the alert rate at your threshold is half a number.


More: https://flyvolo.ai/llms.txt