GenAI’s diagnostic capabilities comparable to non-specialist doctors: Study
A new research on Friday showed that generative AI’s (GenAI) diagnostic capabilities are comparable to non-specialist doctors.
The use of generative AI for diagnostics has attracted attention in the medical field and many research papers have been published on this topic.
However, because the evaluation criteria were different for each study, a comprehensive analysis was needed to determine the extent AI could be used in actual medical settings and what advantages it featured in comparison to doctors.
A research group led by Dr Hirotaka Takita and Associate Professor Daiju Ueda at Osaka Metropolitan University’s Graduate School of Medicine conducted a meta-analysis of generative AI’s diagnostic capabilities, using 83 research papers published between June 2018 and June 2024, that covered a wide range of medical specialties.
Of the large language models (LLMs) that were analysed, ChatGPT was the most commonly studied.
The comparative evaluation revealed that medical specialists had a 15.8 per cent higher diagnostic accuracy than generative AI, according to the study published in the journal npj Digital Medicine.
The average diagnostic accuracy of generative AI was 52.1 per cent, with the latest models of generative AI sometimes showing accuracy on par with non-specialist doctors.
“This research shows that generative AI’s diagnostic capabilities are comparable to non-specialist doctors. It could be used in medical education to support non-specialist doctors and assist in diagnostics in areas with limited medical resources,” said Dr Takita.
“Further research, such as evaluations in more complex clinical scenarios, performance evaluations using actual medical records, improving the transparency of AI decision-making, and verification in diverse patient groups, is needed to verify AI’s capabilities,” according to researchers.
Established in Osaka as one of the largest public universities in Japan, Osaka Metropolitan University said it is committed to shaping the future of society through “Convergence of Knowledge” and the promotion of world-class research.
