The Risks of Generative AI
Generative AI: A Risky Proposition for Healthcare
Generative AI, while powerful, has shown significant shortcomings in healthcare applications. A 2024 study by the University of Massachusetts Amherst and Mendel uncovered alarming rates of hallucinations in medical summaries generated by frontier AI models.
Key Findings:
-
Nearly all summaries analyzed contained hallucinations.
-
Inconsistencies included:
-
327 medical event errors in GPT-4o summaries.
-
271 medical event errors in Llama-3 summaries.
-
Incorrect reasoning and chronological inconsistencies were common.
-
The most frequent hallucinations were related to symptoms, diagnoses, and medicinal instructions, posing serious risks such as misdiagnosis or incorrect treatment.