3415 - Cancer Treatment Information Differences by Bilingual Prompting in Large Language Model Chatbots

J. Mora¹, S. Chen², R. H. Mak³, and D. S. Bitterman³; ¹Harvard Radiation Oncology Program, Boston, MA, ²Department of Radiation Oncology, Brigham and Women’s Hospital/Dana-Farber Cancer Institute, Boston, MA, ³Department of Radiation Oncology, Dana-Farber Cancer Institute/Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

Purpose/Objective(s): There is significant excitement about the potential of large language models (LLMs) to improve health literacy via multilingual question-answering, yet LLMs are predominantly trained on English data. It is unknown how performance varies when accessed by patients who speak different languages. We assessed the generalizability of LLM performance in providing cancer treatment information in English and Spanish. Materials/

Methods: 104 general questions about treatments for prostate, breast, and lung cancers of different disease staging were written in English and translated into Spanish by a bilingual oncologist. GPT3.5-turbo-0613 was prompted with the questions on the same day via API. Quality concordance of responses with NCCN 2021 guidelines (based on GPT’s knowledge cut-off at that time) were scored by a bilingual oncologist using pre-specified criteria (Table). Mann-Whitney U- and Fisher’s exact tests were used to compare scores and frequency of key treatment terms, respectively.
Results: All LLM responses were provided in the same language as the question, despite no explicit instruction on response language. All LLM responses included at least 1 treatment recommendation. The Table shows the scored results overall, and per diagnosis. Clinical trials were recommended in 12/104 English and 1/104 Spanish responses (p<0.01). Specific radiotherapy techniques such as 3DCRT, IMRT, SBRT, and proton therapy were mentioned in 8/104 English and 3/104 Spanish responses (p=0.21). Palliative/supportive care was mentioned in 23/104 English and 6/104 Spanish responses (p<0.01).
Conclusion: The same cancer information question, when asked in a different language, could elicit meaningfully different responses from LLMs. Importantly, asking a question in Spanish led to responses that were less likely to be guideline-concordant or introduce clinical trials/palliative approaches to treatment. This highlights the need for robust multilingual evaluation to harness the promise of LLMs in improving health literacy while avoiding perpetuation of health disparities in vulnerable populations.

Abstract 3415 – Table 1

Criteria	All – English n=104	All – Spanish n=104	p-value*	Prostate – English n=32	Prostate – Spanish n=32	Breast – English n=20	Breast – Spanish n=20	Lung – English n=52	Lung – Spanish n=52
Are the recommended treatments in accordance with NCCN?			0.07
- All - Some but not all - None		68 (65%) 36 (35%) 0		18 (56%) 14 (44%) 0	18 (56%) 14 (44%) 0	18 (90%) 2 (10%) 0	16 (80%) 4 (20%) 0	44 (85%) 8 (15%) 0	34 (65%) 18 (35%) 0
Of the recommended treatments, are any correct in their entirety per NCCN?			0.04
- None - At least 1 - N/A	0 24 (23%) 80 (77%)	1 (1%) 36 (35%) 67 (64%)		0 14 (44%) 18 (56%)	0 14 (44%) 18 (56%)	0 2 (10%) 18 (90%)	0 4 (20%) 16 (80%)	0 8 (15%) 44 (85%)	1 (2%) 18 (35%) 33 (63%)
Are any recommended treatments hallucinated?			0.02
- None - At least one	81 (78%) 23 (22%)	93 (89%) 11 (11%)		22 (69%) 10 (31%)	27 (84%) 5 (16%)	15 (75%) 5 (25%)	18 (90%) 2 (10%)	44 (85%) 8 (15%)	48 (92%) 4 (8%)
*Statistics only calculated overall given small disease subsets