How we know it works, and what is not proven yet
Detection is plain code, so it can be tested like code. The test ward has an answer key the engine is not allowed to read, built to include the cases that fool a careless rule: values just under a threshold and patients whose chronic disease mimics an acute one.
75/75
Planted gaps caught
Right condition and right gap type, on the development ward.
0
False alarms
Across 125 charts that should raise nothing.
75/75
Queries pass the rules check
ACDIS/AHIMA 2022: no leading wording, no billing talk, no invented numbers.
2,250
Browser = engine cases
This site's rules check gives the same answer as the Python one on every case.
Development ward, by test group
200 synthetic inpatient stays.
| Group | What a correct engine does | Charts | Correct |
|---|---|---|---|
| Confounder | Chronic kidney disease, high but stable creatinine: must stay quiet | 15 | 15 |
| Correctly documented | Evidence and notes agree: must stay quiet | 40 | 40 |
| Missing diagnosis | Evidence met, never documented: must flag it | 50 | 50 |
| Near-miss | Just below the published threshold: must stay quiet | 30 | 30 |
| Routine admission | Nothing to find: must stay quiet | 40 | 40 |
| Unsupported diagnosis | Documented, evidence absent: must flag it | 25 | 25 |
Why the test can be trusted
- Two separate programs. One writes the patients and the answer key; the detection rules were written separately and are blocked, by an automated test, from importing the generator or reading the key.
- Hard cases on purpose. Near-misses and confounders are where simple rules raise false alarms, and false alarms are what make clinicians ignore a tool.
- A sealed held-out ward. 100 more stays that no rule was tuned on. It can be scored once; a second attempt is refused, so the final number cannot be massaged.
The clinical rules, and where each threshold comes from
Every flag on a chart lists the exact values and hours that met these rules.
| Condition | Evidence the rule requires | Source |
|---|---|---|
| Sepsis | Suspected infection (blood cultures and antibiotics) plus a SOFA rise of at least 2: MAP < 70 mmHg, platelets < 150 K/uL, bilirubin ≥ 1.2 mg/dL, creatinine rise from the patient's own low | Sepsis-3 (Singer et al., JAMA 2016) |
| Acute kidney injury | Creatinine up ≥ 0.3 mg/dL within 48 h, or to ≥ 1.5x an earlier value within 7 days. A stable high baseline (CKD) does not count | KDIGO 2012 AKI guideline |
| Hyponatremia | Sodium < 135 mmol/L that was clinically significant: treated (fluid restriction, salt tablets, tolvaptan, hypertonic saline) or checked ≥ 4 times in 48 h | Serum sodium < 135 mmol/L (standard definition) |
| Acute blood loss anemia | Hemoglobin drop ≥ 2 g/dL with a documented bleeding source or a transfusion | Common CDI clinical indicators |
| Acute hypoxic respiratory failure | At least 2 of: SpO2 < 91% on room air, PaO2 < 60 mmHg, respiratory rate > 28/min; plus oxygen therapy | Common CDI clinical indicators |
| Acute systolic heart failure | LVEF ≤ 40%, BNP > 400 pg/mL and an IV loop diuretic | 2022 AHA/ACC/HFSA guideline; ESC BNP guidance |
Not proven yet
- Messy notes. These scores use notes written from templates, with clean wording. Real notes hedge, abbreviate and copy forward. The next test uses LLM-written notes with that variety, scored on the same answer key.
- The held-out ward. Not scored yet; it is run once, after the rules are final.
- Real charts. All patients are synthetic. The FHIR reader works on Epic-format data, but no rule has been validated on real records.
- The physician. Answers here come from a rule-based simulator, not clinicians. Billing weights cover the 16 MS-DRGs this ward uses (CMS FY2027, v44).
See it on a single patient: D0011, sepsis missed on a urinary tract infection admission.