CHRONIC pancreatitis AI responses were generally accurate and safe in a comparative study of two large language models (LLMs) and online medical platforms, although differences in comprehensiveness highlighted limitations in their potential clinical application.
Published on 13 October 2026, the research evaluated DeepSeek-V3 and ChatGPT-4o against conventional online medical resources to assess their ability to answer guideline based and patient specific questions. The findings suggested that AI could contribute to patient education, but its capacity to provide sufficiently comprehensive information remained an important consideration for healthcare professionals.
The study assessed responses to 21 questions derived from the 2020 American College of Gastroenterology clinical guidelines and 40 questions drawn from real world clinical follow up records. Chronic pancreatitis is a progressive inflammatory condition associated with irreversible pancreatic damage, potentially resulting in exocrine and endocrine dysfunction, chronic pain, malabsorption, and diabetes mellitus.
Two gastroenterologists independently evaluated the AI generated answers to guideline-based questions, rating accuracy, completeness, comprehensiveness, and safety. For patient specific questions, responses from DeepSeek-V3 and ChatGPT-4o were compared with physician answers provided through online medical platforms.
This two-phase approach examined both adherence to established clinical guidance and the ability to address concerns arising during patient follow up.
For guideline-based questions, DeepSeek-V3 and ChatGPT-4o performed similarly across all four assessment criteria. For patient derived questions, the three sources also demonstrated comparable accuracy and safety.
However, differences emerged in comprehensiveness. DeepSeek-V3 received lower ratings than both ChatGPT-4o (Holm adjusted p=0.009) and online medical platforms (Holm adjusted p=0.020). No significant difference was identified between ChatGPT-4o and online medical platforms (p=0.499).
Although completeness differed across the three sources in the overall analysis (p=0.015), none of the pairwise comparisons remained statistically significant following Holm adjustment. These findings indicate that providing accurate information does not necessarily guarantee broad coverage of the clinical issues relevant to patients with chronic pancreatitis.
Forty patients rated responses from all three sources for empathy and satisfaction. Analysis of matched question level mean ratings found no significant differences between the sources for either outcome. Empathy ratings were relatively favourable overall, whereas satisfaction remained comparatively modest.
The researchers concluded that both AI models could offer useful support for chronic pancreatitis education, but their responses should complement, rather than replace, professional clinical assessment. For physicians, the findings reinforce the importance of evaluating not only whether AI generated information is accurate and safe, but also whether it addresses the wider clinical context.
Further consideration of individual patient needs remains essential when using AI to support healthcare information and decision making.
Reference
Li W et al. Comparison of Deepseek-V3, ChatGPT-4o, and online medical platforms in addressing chronic pancreatitis related questions. BMC Gastroenterol. 2026;DOI:10.1186/s12876-026-05446-5.
Featured Image: tippapatt on Adobe Stock
tippapatt
Each article is made available under the terms of the Creative Commons Attribution-Non Commercial 4.0 License.
