CAMEROON, October 4 — In a recent study published in Discover Artificial Intelligence, four major AI chatbots were evaluated on their ability to provide safety warnings when answering patient questions about pregabalin, a medication used for various conditions. The study, conducted by researchers from Burkina Faso, Niger, and Cameroon, found that all four models failed to include required safety warnings in a significant number of cases.
The chatbots, including ChatGPT-4o, Microsoft Copilot, Gemini 2.0 Flash, and Claude 3.5 Sonnet, were tested using 15 real patient questions drawn from the official NHS pregabalin FAQ. Each question carried an explicit instruction to include a warning if any danger existed.
A multidisciplinary panel of a rheumatologist, neurologist, and pharmacologist, each with at least five years of clinical experience, judged all 15 questions to carry a clinically relevant safety concern requiring a warning. Despite this, warning compliance scores ranged from 0.467 for ChatGPT-4o and Gemini 2.0 Flash to 0.733 for Claude 3.5 Sonnet. This means the models failed to include required safety warnings in 27% to 53% of cases.
Factual accuracy scores were higher, with ChatGPT-4o, Gemini 2.0 Flash, and Copilot scoring 0.933 and Claude 3.5 Sonnet scoring 0.867. Clinical relevance scores ranged from 0.867 to 1.000.
The researchers noted that overlapping confidence intervals meant no model could claim reliable superiority, and the single-query-per-model design did not account for stochastic variability in LLM outputs. Error analysis showed oversimplification accounted for 56.3% of all recorded errors. The study followed the TRIPOD-LLM reporting guideline and used Fleiss’ kappa and intraclass correlation coefficients to assess inter-rater reliability.
Pregabalin, a structural analogue of GABA that binds to the alpha-2-delta-1 subunit of voltage-gated calcium channels, has seen global gabapentinoid sales rise 114.5% between 2012 and 2022 and 180.9% in low- and middle-income countries. Prescribing rates reached 7.2 prescriptions per 100 patients in 2019, heightening concerns about off-label use and polypharmacy risks. As we continue to rely on AI for healthcare information, it’s crucial that these models can accurately and safely provide the information patients need.
The study’s findings underscore the need for ongoing research and improvement in this area.
What’s Next:
Researchers are calling for more studies to assess the performance of AI chatbots in various healthcare scenarios and to identify strategies for improving their safety and accuracy. As AI continues to evolve, it’s essential that we ensure these tools are not only powerful but also trustworthy.
