3145 - An LLM-Based Framework for Zero-Shot De-Identifying Flexible Text Data in Protected Health Information Enabling Potential Risk-Informed Patient Safety

C. W. Chang¹, M. Hu¹, B. Ghavidel¹, J. F. Wynne¹, R. L. J. Qiu¹, M. Washington¹, O. Kayode¹, W. G. Chin¹, K. Yang², J. G. Scott², A. B. Patel Jr¹, and X. Yang¹; ¹Department of Radiation Oncology, Winship Cancer Institute of Emory University, Atlanta, GA, ²Department of Radiation Oncology, Taussig Cancer Center, Cleveland Clinic, Cleveland, OH

Purpose/Objective(s): Safety is an abstract concept that cannot be directly observed, and the Radiation Oncology Incident Learning System (RO-ILS) is designed as a surrogate to enable risk-informed safety assessment and management (RISAM) for enhancing patient safety measures in radiotherapy. Establishing a robust patient safety learning system necessitates assimilating all available, relevant, and adequately evaluated data (ARAED), including incidents from multi-institutional sources. However, HIPAA mandates that any shared medical information must not contain Protected Health Information (PHI). This work aims to develop an artificial intelligence framework leveraging state-of-the-art Large Language Models (LLMs) to effectively and efficiently de-identify PHI, potentially facilitating data-driven solutions for RISAM. Materials/

Methods: LLM features interpretability, allowing us to utilize prompts to query information with semantic reasoning. We integrated a generative pre-trained transformer (GPT)-based open source LLM (LLaMa2-70B) with HIPAA identifiers into the proposed RISAM framework for de-identifying electronic PHI. The LLaMa2 with 70 billion pre-trained parameters based on 2 trillion tokens underwent validation by the institutional safety database with 1222 incidents happened between 2015 and 2023. The evaluation encompassed two objectives: 1) assessing model flexibility for potential multi-institutional collaboration, and 2) evaluating model robustness for semantic reasoning across data from different hospitals. A zero-shot testing was conducted using unstructured text data. The de-identified results were comprehensively reviewed by the institutional quality committee to ensure accuracy in fulfilling HIPAA guidelines.
Results: The evaluated GPT-based LLM achieved the zero-shot accuracy of 95.2% in de-identification. After the review of the internal quality committee, most mis-redacted content included unstructured dates without years (MM/DD) and ages followed by acronym of years (YO). Major HIPAA identifiers were redacted to protect patients privacy, including but not limited to names, dates, and medical record numbers.
Conclusion: LLaMa2-70B demonstrated highly accurate de-identification capabilities for unstructured text-based incident reports. This validation is crucial for the continued development of the RISAM framework, which aims to effectively analyze trends and root causes of medical incidents, utilizing the findings as medical education resources. A robust de-identification method can facilitate the development of a trustworthy LLM-based RISAM framework by assimilating ARAED. Future investigations will focus on deploying a radiation oncology-specific LLM-based safety framework as a virtual safety assistant in clinical settings, enabling medical staff to query solutions during emergent safety incidents.