1. Anderson and colleagues compared clinical staff response times to patient messages with NLP labeling versus without NLP.
2. NLP shortened the time required to respond to new patient messages and to complete patient conversations.
Evidence Rating: Level 2 (Good)
Study Summary:
Patients are increasingly using EHR messaging portals for care, but messages often get routed manually through a central pool before reaching the right staff, causing delays. To address this, Anderson and colleagues developed an NLP model to categorize incoming messages into common themes, aiming to speed up response times. The model was trained on 40,000 EHR messages and sorted messages into five categories: urgent, clinician, refill, schedule, or form. After deployment in a clinical setting, the response times of NLP-routed messages were compared to a similar group of manually routed messages. Key measures included time to first staff interaction, time to complete the conversation, and total messages exchanged. Results showed that NLP-routed messages reached healthcare staff faster and conversations were completed more quickly. The NLP system also consistently categorized messages accurately. This study demonstrates that integrating an NLP classifier within EHRs can improve response times and reduce the messaging workload for healthcare staff.
In-Depth \[Prospective Cohort]:
The NLP model was developed using a dataset of 40,000 EHR messages from adult patients, with each message annotated by a clinician into one of five categories: urgent, clinician, refill, schedule, or form. After development, the model was implemented across four outpatient sites. The intervention group had messages automatically routed by the NLP, while the control group consisted of a parallel set of unrouted messages. Both groups’ messages were collected from the same sites during the same two-week period, following identical inclusion and exclusion criteria.
Primary outcomes compared were the time from message initiation to first healthcare staff interaction (including reads, forwards, or replies), time from initiation to conversation completion, and the total number of message interactions by staff. Secondary outcomes assessed the NLP’s precision, recall, and accuracy in labeling messages.
Results showed that the intervention group experienced a median 1-hour faster initial response time (95% CI: −1.42 to −0.5 hours) and a 22.5-hour shorter median time to complete conversations (95% CI: −36.3 to −17.7 hours). Staff in the NLP-routed group also handled fewer total message interactions, with a median reduction of 2 interactions (95% CI: −2.9 to −1.4). The NLP demonstrated precision, recall, and accuracy rates exceeding 95% across all five categories.
Overall, this study confirmed that using an NLP classifier within the EHR can improve operational efficiency and reduce administrative workload for healthcare teams.
