3436 - Medical Accuracy of Cancer Radiotherapy-Related ChatGPT AI Outputs in English and Spanish

P. Sharma¹, R. Yoder¹, X. Shen², J. P. Einck³, K. L. Rhodes-Stark³, G. N. Gan³, J. C. Shiao³, D. Cunningham³, Y. S. Butler-Xu³, A. Tejwani³, R. C. Chen², and S. R. Stecklein³; ¹The University of Kansas Medical Center, Kansas City, KS, ²Department of Radiation Oncology, University of Kansas Medical Center, Kansas City, KS, ³Department of Radiation Oncology, The University of Kansas Medical Center, Kansas City, KS

Purpose/Objective(s): Large language models provide patients with new avenues to query medical questions. The medical accuracy of large language model outputs and concordance of these outputs in more than one language are not well established. This study assessed medical accuracy and concordance of ChatGPT recommendations in English and Spanish regarding cancer radiotherapy. Materials/

Methods: Five radiotherapy case scenarios were written in English, one for each of the following cancers: prostate, lung, pancreas, breast, and brain. Scenarios were translated into Spanish by a medical interpreter and were inputted into ChatGPT 3.5 in English and Spanish separately. Spanish responses were translated back into English by the same medical interpreter. ChatGPT treatment recommendations were rated by nine board-certified radiation oncologists for medical accuracy and completeness on a scale of 1-4 (1: excellent, 2: good, 3: fair, 4: poor). Radiation oncologists were blinded as to which recommendations were in response to English vs Spanish input into ChatGPT. Scores were compared using the Wilcoxon matched pairs signed rank test.
Results: 35 ratings were obtained for both accuracy and completeness (prostate = 8; lung = 8; pancreas = 6; breast = 6; brain = 7). Median medical accuracy across the five scenarios for English output was 2.0 (range 1-4) and for Spanish output was also 2.0 (range 1-4) (P = 0.892). Median completeness for English and Spanish outputs was 2.0 (range 1-4) and 2.0 (range 1-4), respectively (P = 0.550). Median accuracy and completeness for each scenario and all scenarios combined are shown in the Table. For the lung cancer scenario, English output trended towards more accurate (P = 0.125) and complete (P = 0.063) compared to Spanish output. English and Spanish outputs had similar accuracy and completeness ratings for the other four case scenarios. On repeated measures ANOVA analysis there was no evidence of significant variability by cancer type or rater for accuracy (P=0.330 and P=0.980, respectively) or completeness (P=0.267 and P=0.476, respectively).
Conclusion: Radiotherapy recommendations provided by ChatGPT were moderately accurate and complete. ChatGPT 3.5 can query information only until 2022 and thus may not capture recent advances and changes in standard-of-care treatment, leading to lower ratings for certain scenarios. The majority of medical studies are published in English and there may be errors in translating information into other languages. These important questions should be studied further. Abstract 3436 – Table 1

Cancer Type	Medical Accuracy; Median (Range)			Completeness; Median (Range)
	English	Spanish	P value	English	Spanish	P value
Prostate	2.0 (1-3)	1.5 (1-2)	0.375	2.0 (1-3)	2.0 (1-3)	0.750
Lung	2.0 (1-3)	2.5 (1-4)	0.125	1.5 (1-3)	2.0 (1-4)	0.063
Pancreas	3.0 (1-4)	2.0 (1-4)	0.625	3.0 (1-4)	2.5 (1-4)	0.625
Breast	2.0 (1-3)	2.5 (1-4)	0.500	2.0 (1-3)	3.0 (1-4)	0.500
Brain	2.0 (1-3)	1.0 (1-3)	1.000	2.0 (1-3)	2.0 (1-3)	1.000
All	2.0 (1-4)	2.0 (1-4)	0.892	2.0 (1-4)	2.0 (1-4)	0.550