3427 - Evaluating ChatGPTs Competency in Radiation Oncology: A Comprehensive Assessment across Clinical Scenarios

S. Ramadan¹, A. Mutsaers¹, G. S. Bauman¹, V. Velker¹, B. Ahmad¹, A. J. Arifin², T. Nguyen¹, D. A. Palma¹, and C. D. Goodman¹; ¹Department of Radiation Oncology, London Health Sciences Centre, London, ON, Canada, ²London Health Sciences Centre, Western University, London, ON, Canada

Purpose/Objective(s): The advent of artificial intelligence (AI) and machine learning tools presents a unique opportunity for enhancing clinical decision-making in radiation oncology (RO). However, AI models can be subject to biases and misinformation. This study aims to evaluate the competency of ChatGPT, an advanced generative AI language model, in interpreting clinical scenarios and assessing its oncology knowledge. Materials/

Methods: A series of clinical cases were designed by the authors covering 11 common disease sites. Questions were grouped into the following domains: epidemiology, staging and workup, clinical management, treatment planning, cancer biology, physics, and surveillance. Royal College-certified ROs reviewed cases and provided benchmark solutions. ChatGPT was prompted to answer questions in a “factual and scientific” manner. ROs scored AI responses on 3 criteria: conciseness (focused answers), completeness (addressing all aspects of the question), and correctness (answer aligns with expert opinion), using a standardized rubric. Scores ranged from 0-5 for each domain for a total possible score of 15. Cumulative scores were analyzed and compared across domains utilizing a one-way ANOVA with a post-hoc test.
Results: Across 11 cases, 182 questions were answered with a total AI score of 2317/2730 (84%). Scores by criteria were: completeness (79%, range: 70-99%), conciseness (92%, range: 83 – 99%), and correctness (81%, range: 72 – 92%). AI performed best in domains of epidemiology (93%) and cancer biology (93%) and reasonably in staging and workup (89%), physics (86%) and surveillance (82%). Weaker domains included treatment planning (78%) and clinical management (81%). Statistical differences in scores across domains were driven by variations in the completeness (p<0.01) and correctness (p=0.04) criteria, whereas conciseness scored universally high (p=0.91). Allowing for inter-rater differences in absolute scores, these trends were consistent across disease sites.
Conclusion: A robust analysis of Chat GPT was performed utilizing custom cases benchmarked against content experts. ChatGPT showed potential as a supplementary tool in radiation oncology, demonstrating a high degree of accuracy in several oncologic domains. However, this study highlights a significant limitation with incorrect and incomplete answers in complex cases. Future work aims to analyze the difference between human and AI responses while investigating methodologies to improve AI accuracy. Abstract 3427 – Table 1: Chat GPT 4.0 scores on all 182 questions

Domain	Completeness	Conciseness	Correctness	Overall
Epidemiology	99%	92%	87%	93%
Staging and Workup	82%	99%	85%	89%
Clinical Management	74%	91%	79%	81%
Treatment Planning	70%	91%	72%	78%
Cancer Biology	94%	93%	92%	93%
Physics	80%	96%	83%	86%
Surveillance	82%	83%	82%	82%
Total Score	79%	92%	81%	84%