3405 - Evaluation of Deep Learning-Based Abdominal Organ Segmentation Models: Assessing Robustness across Non-Surgery to Organ Resection Surgery Patients

W. Liao¹, and X. Luo²; ¹department of Radiation Oncology, Radiation Oncology Key Laboratory of Sichuan Province, Sichuan Clinical Research Center for Cancer, Sichuan Cancer Hospital& Institute. Sichuan Cancer Center, Affiliated Cancer Hospital of University of Electronic Science, Chengdu, China, ²School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China

Purpose/Objective(s): In recent years, deep learning-based abdominal organ segmentation methods have shown considerable promise. However, there remains a gap in evaluating these methods for patients with irregular anatomies, such as those who have undergone organ resection. This work aims to comprehensive evaluate the segmentation robustness of state-of-the-art methods on clinical challenging cases. Materials/

Methods: A total of 413 patients from hospital A were retrospectively collected for model development and evaluation, where 317 patients without receiving any surgery (Set1), 22 patients received surgery but preserve partial organ (Set2) and 74 patients received organ resection surgery (Set3). A senior expert manually delineated 19 organs for each patient according to their radiology report and the RTOG guideline. We investigated several state-of-the-art methods performance and robustness directly, including nnUNet, 3D-UXNet and SwinUNETR. Firstly, all these methods were trained and tested on the Set1 with the split of 8:2 and then they were directly evaluated on the Set2 and Set3. The mean Dice similarity coefficient (DSC) and normalized surface dice (NSD) were used to measure the model performance. In addition, the mean false-positive rate (FPR) of resection organs was employed to measure the model robustness.
Results: In the internal testing dataset (Set1) and two surgery resection datasets (Set2 and Set3), the best DSC was 86.7%, 84.1% and 80.6%, respectively. And the best NSD was 86.9%, 83.7% and 79.9%, respectively. Compared with the non-surgery cohort, with surgery resection cohorts’ performance dropped significantly in terms of DSC and NSD (all p-values < 0.05). In addition, the results of Set3 in terms of FPR were consistently larger than 60%, the best FPR of 63% was achieved by nnUNet. The comprehensive study indicated that state-of-the-art deep learning-based methods also cannot obtain promising segmentation results for surgery resection patients.
Conclusion: This study comprehensively evaluated the state-of-the-art deep learning-based segmentation methods on clinically challenging cases and discovered they cannot handle patients with resection surgery well. This study raised a new research direction that pays more attention to the robustness of clinically challenging cases.

Abstract 3405 – Table 1

Methods	DSC(Set1/Set2/Set3)	NSD(Set1/Set2/Set3)	FPR(Set3)
nnUNet	86.7%/84.1%*/80.6%*	86.9%/83.7%*/79.9%*	63%
3D-UXNet	84.3%/84.3%/76.3%*	83.9%/83.3%/74.4%*	82%
SwinUNETR	83.2%/83.0%/74.9%*	82.4%/81.7%/73.0%*	89%