3434 - Independent Validation and Performance Comparison of Published Radiation Pneumonitis NTCP Models for IMRT, Immunotherapy and Proton Therapy Cohorts

R. M. Salazar¹, S. S. Nair¹, A. O. Leone¹, T. Xu², Z. Liao², L. E. Court¹, and J. Niedzielski¹; ¹Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, ²Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX

Purpose/Objective(s): To test the performance of 8 different published normal tissue complication probability (NTCP) models for radiation pneumonitis (RP), each trained on its own external data, on 3 internal patient cohorts treated with different modalities for non-small cell lung cancer (NSCLC). Materials/

Methods: We analyzed 216 patients receiving IMRT only, 162 proton therapy (PT), and 108 IMRT plus immunotherapy (IO) at our institution. 8 logistic regression/LKB models for RP CTCAE v5.0 grade =2 were evaluated. All were applicable to IMRT cohorts and consisted of QUANTEC, updated QUANTEC, Appelt et al., updated Appelt, Huang et al., Tucker et al., Bradley et al., and vonReibnitz et al. Models were assessed using accuracy, area under the receiver operating characteristic curve (AUC), and calibration slope. By building 1000 bootstraps per dataset, 95% confidence intervals were calculated for AUC. Differences in the bootstrap distributions were tested with the Wilcoxon signed-rank test. A likelihood ratio closed testing procedure (CTP) determined if a model could significantly (p<0.05) improve its fit to the data via recalibration.
Results: Model performance varied across treatment modalities, with accuracy/AUC ranges of 0.69-0.79/0.65-0.73 (IMRT), 0.63-0.82/0.60-0.79 (proton), and 0.61-0.70/0.57-0.66 (IO). The QUANTEC model performed the best on the IMRT cohort, with no change needed to improve its fit; the updated Appelt model yielded the best results for the PT data, with no change needed to improve its fit. The best model performance on the IO cohort was significantly (p<2.2x10^-16) inferior to the other groups.
Conclusion: Our findings reveal large performance variability among published models and across treatment modalities. The QUANTEC and updated Appelt models showed robust performance on our IMRT and PT data. All models underperformed for the IO cohort, indicating a need for developing new NTCP models for this group. Abbreviations: MLD = mean lung dose, D10H = minimum dose to the hottest 10% of the heart, DmaxH = maximum heart dose, D65L = minimum dose to the hottest 65% of the lung, TxDays = treatment days, IMRT = IMRT only, PT = Proton Therapy, IO = IMRT plus immunotherapy, CS = Calibration Slope, CI = Confidence Interval, NC = No Change, R = Recalibrate Abstract 3434 – Table 1: Features and metrics for the top 2, median, and worst performing models

Model (Features)	IMRT			PT			IO
	AUC (95% CI)	CTP	CS	AUC (95% CI)	CTP	CS	AUC (95% CI)	CTP	CS
QUANTEC (MLD)	0.72 (0.64 – 0.82)	NC	1.0±0.3	0.73 (0.65 – 0.81)	R	1.8±0.5	0.65 (0.53 – 0.77)	R	1.8±0.7
Updated Appelt (MLD, Age, Smoking Status)	0.73 (0.65 – 0.81)	R	0.9±0.2	0.79 (0.71 – 0.87)	NC	1.1±0.3	0.66 (0.52 – 0.80)	R	0.4±0.2
Huang (MLD, D10H)	0.70 (0.62 – 0.78)	R	1.4±0.3	0.66 (0.56 – 0.76)	R	0.9±0.4	0.66 (0.54 – 0.78)	R	1.1±0.5
vonReibnitz (DmaxH, D65L, TxDays)	0.67 (0.59 – 0.75)	R	0.8±0.3	0.55 (0.45 – 0.65)	NC	0.6±0.5	0.57 (0.41 – 0.73)	R	0.3±0.3