3452 - Fine-Tuning Large Language Models for Radiation Oncology, A Specialized Health Care Domain

P. Wang¹, Z. Liu², Y. Li², J. Holmes¹, P. Shu², L. Zhang¹, X. Li³, Q. Li³, S. A. Vora¹, S. H. Patel¹, T. T. W. Sio¹, T. Liu², and W. Liu¹; ¹Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, ²School of Computing, University of Georgia, Athens, GA, ³Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA

Purpose/Objective(s): To answer the question of whether fine-tuned large language models (LLM) perform better than vanilla LLMs on radiation oncology clinical tasks and to compare LLMs’ performance on three clinically important specific tasks: Task (1) treatment regimen generation, Task (2) treatment modality selection (proton, photon, brachy, or electron), and Task (3) ICD-10 code prediction. Materials/

Methods: 115,724 patient cases were extracted from the database of the Department of Radiation Oncology. Cases where each patient had a single diagnostic record and a clearly identifiable primary treatment plan were selected to ensure high-quality data. Processing and manual labeling of the patient diagnosis, treatment plan, treatment modality, and ICD-10 code were conducted, resulting in 7,903 pairs of diagnosis–treatment regimen, 7,903 pairs of diagnosis–treatment modality, and 7,177 pairs of diagnosis–ICD-10 code data for the fine-tuning for these three tasks. Each data pair was used to construct a sample consisting of a prompt and an answer for fine-tuning. Open-source LLaMA2-7B and Mistral-7B models were utilized for the supervised fine-tuning with Low-Rank Approximations method (LoRA) for three clinically important tasks on Nvidia A100 80GB GPUs. ROUGE-1 score which measures the overlap of unigrams was used for the evaluation of Task (1), while accuracy values were used for Task (2) and Task (3). Temperature of LLMs was scanned from 0.1 to 1 with a step size of 0.1 and one-sided Wilcoxon signed-rank tests were used across all the statistical analyses in the results.
Results: Fine-tuned LLMs outperformed vanilla LLMs across all tasks: LLaMA2-7B – Task (1) ROUGE-1 0.531 vs. 0.075 (p-value = 0.001), Task (2) accuracy 0.705 vs. 0.499 (p-value = 0.001), Task (3) accuracy 0.642 vs. 0.180 (p-value = 0.001), and Mistral-7B – Task (1) ROUGE-1 0.545 vs. 0.058 (p-value = 0.001), Task (2) accuracy 0.722 vs. 0.367 (p-value = 0.001), Task (3) accuracy 0.737 vs. 0.082 (p-value = 0.001). Fine-tuned Mistral-7B outperformed fine-tuned LLaMA2-7B in all tasks overall (p-value << 0.001), while vanilla LLaMA2-7B outperformed vanilla Mistral-7B in all tasks overall (p-value << 0.001).
Conclusion: LLMs, fine-tuned with radiation oncology domain data, led to statistically significant improvements in the performance of three clinically important tasks, compared with the vanilla LLMs. Our study demonstrated the great potential of fine-tuned LLMs in radiation oncology and contributed useful results and references for LLM development, fine-tuning, and applications in the clinical tasks of the radiation oncology and healthcare domain. Abstract 3452 – Table 1

Fine-tuned (FT) model	vs.	Vanilla model	Tasks	p-value	Confidence Interval (CI)
LLaMA2 FT	>	LLaMA2	(1), (2), (3)	0.001	0.05
			Overall	<< 0.001
Mistral FT	>	Mistral	(1), (2), (3)	0.001
			Overall	<< 0.001
LLaMA2	vs.	Mistral	Tasks	p-value	CI
LLaMA2	>	Mistral	(1), (2), (3)	0.001	0.05
			Overall	<< 0.001
LLaMA2 FT	<	Mistral FT	(1), (2), (3)	0.001, 0.016, 0.001
			Overall	<< 0.001