Microsoft AI’s diagnostic system correctly identified 83.2% of complex New England Journal of Medicine cases in a simulated benchmark, compared with 36.1% for generalist physicians who had search and reference tools, according to a study published on Thursday in Science.
The system, the Microsoft AI Diagnostic Orchestrator (MAI-DxO), simulates a panel of physicians that propose diagnoses, choose questions and tests, and challenge one another before committing to an answer. It starts from a brief case summary and must request each piece of history, exam findings or test results step by step, rather than receiving the full case at once.
The benchmark, called SDBench, converts 304 NEJM clinicopathological conference cases and 325 emergency department admissions from the MIMIC-IV-ED database into interactive cases. Physicians were tested on a 56-case held-out set of the NEJM cases. Without reference tools, they averaged 22.3%. On the emergency cases, MAI-DxO scored 80.3% versus 46.8% for physicians, the paper said, as Medical Economics also reported.
Cost results depended on configuration. Without budget limits, the system spent more on tests than physicians did, at $5,431 per NEJM case versus $4,725, according to the study. With budget limits, it reached 80.9% accuracy at $2,241 per case and 79.6% at $1,303.
The researchers said the advantage held, and grew relative to a basic prompt, when they reran the full evaluation with GPT-5.5 without changing the orchestration. Microsoft said it is open-sourcing SDBench for select NEJM cases, and the benchmark code is on GitHub under the MIT License. The NEJM-derived case data is available from NEJM Group under a noncommercial research license.
The authors listed several limitations. The cases were text-only, with no imaging, waveforms or pathology slides. The comparison was with individual generalists, not specialists or multidisciplinary teams, and the authors said they would expect the gap to narrow against matched specialists. Every case had a diagnosed disease, so healthy and benign visits were excluded. The study measured diagnosis only, not treatment, safety or patient outcomes. The benchmark measured reasoning under idealized conditions, not bedside performance, the authors wrote. All authors are Microsoft employees who hold equity in the company.
Next, Microsoft will work with Beth Israel Deaconess Medical Center in Boston on a prospective study in which patients interact with the system directly, the company said. Reuters reported that the work will test the tool’s ability to triage patients in emergency department waiting rooms.
“No patient ever came to see me with a nice summary,” Dominic King, corporate vice president of the health unit at Microsoft AI, told Reuters.
Sources
- Science (Nori et al.): Sequential diagnosis with language models (October 22, 2026)
- Microsoft AI: Building Medical Superintelligence (October 22, 2026)
- Reuters (Lapid), via Investing.com: Microsoft develops AI tool that can diagnose ‘sequentially’ like a doctor (October 22, 2026)
- Medical Economics (Shryock): AI system outperforms physicians on diagnostic accuracy in Microsoft study (October 22, 2026)