Microsoft's AI diagnostic system beats doctors on test cases in Science study

Microsoft AI says its diagnostic system can diagnose conditions up to four times more accurately than human doctors in tests on medical scenarios. The findings were published Thursday in Science. The system, called MAI-DxO or the Diagnostic Orchestrator, uses multiple AI agents that act like a panel of doctors: they question a patient, order tests and debate a diagnosis.

On cases drawn from hundreds of New England Journal of Medicine reports, the system reached about 83% accuracy. Generalist physicians using references scored roughly 36%, and those without references 22%. On emergency department cases from Beth Israel Deaconess Medical Center in Boston, it scored 80% against an average of 47% for physicians. It also tended to reach diagnoses at lower estimated cost.

Microsoft compiled the cases into its own benchmark, SDBench, which it plans to open-source. The system was tested with models from several companies, including OpenAI, Google, Anthropic, Meta and DeepSeek, with its best results coming from OpenAI's GPT-5 and GPT-5.5.

The headline comparison comes with caveats. TechCrunch notes that cases this complex are usually handled by specialists or physician teams, who were not part of the comparison. The system has only been tested on past cases, and Suleyman himself says it needs clinical validation, calling real-world use a leap.

TechCrunch's reading of the study's data is that most of the improvement comes from stronger underlying models. GPT-5 alone scored 78.9% on complex cases; the orchestrator cut cost by about $3,000 per case but added only about four points of accuracy. With GPT-5.5, it raised complex-case accuracy from 72% to nearly 87%, but emergency cases stayed near 85% either way. It helped most with weaker models, such as Gemini 2.5 Flash, which rose from 46.4% to nearly 69%.

The paper also does not test newer models, though the researchers say improvements in model capability do not necessarily diminish the benefits of multiple agents working together. The work shows Microsoft building its own health AI stack as competitors publish similar claims.

Microsoft will begin testing with Beth Israel Deaconess in hospital settings. Patients will receive standard care, and those who consent will also interact with the system, with its recommendations compared against human doctors'. Dominic King says this should get underway in the next couple of months, though neither he nor Suleyman committed to a firm timeline.

Commercialization is possible, but Microsoft has not said when or how. King stressed the high regulatory and compliance requirements in medicine. Suleyman says a healthcare foundation model trained with the Mayo Clinic is planned for release next year.