Is it possible for generative artificial intelligence (AI) to scan clinical notes and accurately and quickly extract pertinent data to aid in patient care or scientific research?
According to recent studies from the Mailman School of Public Health at Columbia University, not yet.
A study conducted from 2019 to 2022 that involved 54,569 ER visits from patients hurt while riding a bicycle, scooter, or other micromobility conveyance. Researchers utilized ChatGPT-4 to read medical records and ascertain whether or not injured riders of bicycles and scooters were wearing helmets.
Can generative artificial intelligence (AI) be used to swiftly and correctly extract relevant data from clinical notes to support scientific study or patient care?
Not yet, per recent research from Columbia University’s Mailman School of Public Health.
A study involving 54,569 ER visits from patients injured while riding a bicycle, scooter, or other micromobility conveyance was carried out between 2019 and 2022. Researchers reviewed medical records using ChatGPT-4 to determine whether or not injured scooter and bicycle users had on helmets.
According to reports, the text string-search-based approach’s LLM only functioned properly when the prompt contained all of the text. Additionally, over the course of five consecutive days, it struggled to replicate its work across trials; it performed better at simulating its hallucinations than its precise work. It had the most trouble reading negated phrases like “w/o helmet” or “unhelmeted” and reporting that the patient wore a helmet.
Written clinical notes, a form of unstructured data, contain a significant amount of medically relevant data that is kept in electronic medical records. Research would greatly benefit from effective methods for reading these notes and extracting information.
At the moment, data from these clinical notes can be retrieved by artificial intelligence (AI)-based techniques like natural language processing or more complex methods like string-matching text search methodology. New LLM, like ChatGPT-4, were supposed to be able to extract data more quickly and accurately.
Professor of epidemiology at Columbia Mailman School and senior author Andrew Rundle, DrPH, stated, “While we see potential efficiency gains in using the generative AI LLM for information extraction tasks, issues of reliability and hallucinations currently limit its utility.”
There were days when ChatGPT-4 was able to extract precise data from the clinical notes when we utilized extremely specific prompts that contained every text string pertaining to helmets. However, the amount of time needed to define and test every word that needed to be in the prompt and ChatGPT-4’s incapacity to consistently reproduce its work suggest that ChatGPT-4 was not yet ready for this assignment.
The most recent study expands on their earlier research on injury prevention strategies for micromobility users, such as scooter, e-bike, and bicycle riders.
“Although wearing a helmet reduces the severity of an injury, information about helmet use is typically hidden in the clinical notes that the attending physician or emergency medical services provider writes in emergency department medical records and incident reports. The lead author of the work and a post-doctoral associate in the Mailman School’s Department of Epidemiology, Kathryn Burford, stated that there is a great need for research to be able to rapidly and reliably access this information.
According to Rundle, “our study investigated the potential of an LLM for information extraction from clinical notes, a rich source of information for health professionals and researchers.”
However, ChatGPT-4 was unable to provide us with data in a dependable manner when we used it.
The study’s results have been published in JAMA Network Open. Nicole G. Itzkowitz from Columbia Mailman School of Public Health, Ashley G. Ortega from Columbia Population Research Center, and Julien O. Teitler from Columbia School of Social Work are co-authors of this work.
