3558 - Improving Academic Productivity or Raising Ethical Concerns? Time-Series Analysis of AI-Assisted Scientific Writing

C. E. Read¹, M. Khazaieli², and J. Xia³; ¹Icahn School of Medicine at Mount Sinai, New York City, NY, ²California University of Science and Medicine, Colton, CA, ³Icahn School of Medicine at Mount Sinai, NEW YORK, NY

Purpose/Objective(s): Generative AIs rise prompts our exploration into AI-assisted scientific writing. By the 1990s, NLP models used statistical approaches to detect textual patterns. OpenAIs GPT-2 debuted in February 2019, and by 2023, ChatGPT 3.5 gained prominence. We aim to offer key insights for navigating the growing role of AI in scientific writing to foster productive discourse and a framework. Our hypothesis: increased generative AI accessibility correlates to a surge in AI-assisted content in academic writing. Materials/Methods: zeroGPT was selected as the generative AI detection software based on its accuracy from published data. A validation study, including 21 journal articles from 1980 and 2000, was conducted to evaluate zeroGPT. A subsequent time-series analysis (2019-2024) was performed on 262 cancer-related, full-length, peer-reviewed research articles from 2 journals (J.1/ J.2) and 207 scientific conference abstracts using zeroGPT. Both journals’ impact factors range between 5 and 8.5 (2019-2024). J.1 is paywalled, and J.2 is open access. Results were categorized using binary and percent classification methods.

Results: In the validation study, the average percentage of AI-assisted writing was 1.2% in 1980 and 12.3% in 2000. This serves as a baseline for the background benchmark. For J.1, average AI content varied from 18.2% (2022) to 22.2% (2021), with all-time maximum of 68.6% (2020). For J.2, average content varied from 28.8% (2019) to 52.2% (2024), with all-time maximum of 94.8% (2023). For the conference abstracts, average content varied from 3.6% (2019) to 12.2% (2021), with all-time maximum of 58.1% (2023).

Conclusion: Journal-specific analysis reveals intriguing trends, notably J.1’s relatively consistent average AI usage (18.2% to 22.2%) and J.2s fluctuation (28.8% to 52.2%). Maximum content as high as 94.8% (J.2, 2023) prompts questions about scientific merit. Inter-journal content differences may relate to the journal access model. Also, at some point, J.1 began mandating a statement on generative AI use; might this relate to the 2022 dip in J.1’s AI content? Ultimately, while generative AI tools can lead to improved communication and a streamlined writing process, risks include erroneous conclusions about data and falsely authoritative tones. Looking ahead, proactive measures such as educational initiatives for effective AI integration should be considered. Our work does not aim to assign responsibility to specific journals or conferences but rather stands as a litmus test to predict the realistic relevance of AI generated content in published scientific articles from 2019-2024. Abstract 3558 – Table 1

Year	No. Articles	J.1 Avg. %	J.1 Max. %	No. Articles	J.2 Avg. %	J.2 Max. %	No. Abstracts	Abstracts Avg. %	Abstracts Max. %
1980	12	1.1	11.9	~	~	~	~	~	~
2000	9	12.3	29.0	~	~	~	~	~	~
2019	29	27.1	63.6	20	28.0	74.1	42	3.6	20.3
2020	25	27.4	68.6	20	37.5	76.6	42	5.8	53.8
2021	20	22.2	68.5	20	39.6	84.7	40	12.2	47.0
2022	26	18.2	36.4	20	41.1	79.1	42	3.9	47.4
2023	20	21.2	61.4	20	38.8	94.8	41	8.5	58.1
2024	22	26.7	61.8	20	52.2	83.5	~	~	~