N. J. Eustace1, K. Man2, T. Lu2, S. Mehdinia2, A. Tam1, C. J. Ladbury1, and Y. R. Li1; 1Department of Radiation Oncology, City of Hope National Medical Center, Duarte, CA, 2Department of Applied AI & Data Science, City of Hope National Medical Center, Duarte, CA
Purpose/Objective(s): Large language models (LLMs) like Generative Pre-trained Transformer 4 (GPT-4) provide an opportunity to utilize an artificial intelligence (AI) framework to extract and interpret data contained within unstructured clinical text to provide an interactive and context-aware understanding of the patients history. HopeLLM is an AI framework specializing in processing oncology clinical notes and providing tools tailored for cancer care, including a conversational interface to query patient medical records. This tool has the potential to accelerate clinical and research workflows by helping researchers rapidly review medical records to abstract key clinical endpoints. We present results on the use of HopeLLM for data abstraction of prostate-specific antigen (PSA) values present in unstructured clinical notes, a first step for determining the biochemical recurrence of prostate cancer. Materials/
Methods: HopeLLM is built around a custom embedding model trained on a corpus of billions of tokens of text drawn from City of Hope clinical notes. We tested the performance of HopeLLM in abstracting PSA values and their corresponding dates from unstructured clinical notes of patients who later developed biochemical recurrence and compared these results to data manually abstracted by physicians. We report on the concordance of PSA values and dates returned from HLM with manual abstraction performed by a clinician and explore the context of non-concordant data. Results: An initial dataset of PSA values was manually abstracted from six patients. 32 unique records of PSA measurements were aligned across manual and HopeLLM abstraction of a health care software electronic medical record. The mean absolute percent error (MAPE) of the abstracted PSA values was 2.9%, and the mean difference of PSA measurement dates was 0.23 months, or less than one week. In addition, HopeLLM surfaced an identified 7 PSA measurements that appeared in the clinical notes but did not appear in the manual abstraction. Conclusion: Overall, manual abstraction is a time-consuming and error-prone process, and we find HopeLLM to quickly and accurately abstract PSA values and dates from clinical text. LLM can streamline data collection and warrants validation on larger datasets and at other institutions.
