Edmonton Zone triage project (EZ Triage)

Validation of an automated triage audit system

Abstract

Background: Canadian Emergency Departments (ED) use the five-point Canadian Triage Acuity Scale (CTAS) to prioritize patients according to acuity. CTAS scores are used to make decisions on patient flow, staffing complement, and funding. Variations in triage can lead to mis-categorization and delayed care for critically ill patients. Edmonton zone quality improvement audits reduced high and low-risk vital sign errors (86% and 78%, respectively), increased between-nurse triage consistency, and levelled triage burden for individual nurses. However, previous audits were time-consuming and required auditors with clinical, data management, and analysis expertise, preventing their wide-scale adoption.

Implementation: This project utilizes the AHS AIW implementation framework. It is the extension of a triage audit and feedback tool that was developed and validated over a three-year window (Cotton et al., 2021). This current project is a scaling-up of the QI audit framework to a Tableau dashboard that will allow ED administrators and educators to efficiently examine individual nurse-level and department triage variation to craft local QI measures to improve triage accuracy.

The newly developed QI tool uses raw EPIC data retrieved from AHS data warehouses. It translates the framework to a python script that duplicates existing Excel QI framework Boolean logic and generates binary output variables. These variables are then loaded into a preformatted Tableau dashboard that displays both department and nurse-level triage variances (trimmed mean and standard deviation) for the following variables: high and low acuity vital sign error rates, triage acuity overrides, and sepsis alerts. The dashboard data (and visualizations) allow users to filter errors by nurse and error type. They include all the relevant triage-associated data and can be used for near real-time monitoring of triage variances or downloaded for additional department-level analyses.

Patient and Family Engagement: Triage has consistently been identified as an area of concern for hospital administrators, clinicians, and patients. Most complaints received by hospitals about ED care are generated at triage. Literature has suggested that triage can be cognitively demanding for nurses, and data have suggested that patients have experienced bias during triage. This project will benefit patients by minimizing errors, ensuring consistent triage, and allowing other sites across the province to engage in similar QI efforts. Because these triage QI efforts require the review of patient-sensitive data, patients were not included in the analyses. 

Evaluation Methods: The output of the triage audit dashboard was assessed in two ways: by soliciting the feedback of clinical audit end-users on the ease of use and operability of the dashboard, and by assessing the reliability of the tool by comparing it to the existing definitions and standard of screening.

The user interface of the triage audit system was assessed and refined soliciting feedback from end-users of the system. Each refinement cycle was accompanied by further feedback. A random 10,000-patient sample was used to perform assess the performance of the tool. The reliability of the dashboard was calculated by the sensitivity, specificity, accuracy, and agreement of expert raters. Between-group comparisons of expert and automated systems to gold-standard manual reviews were performed using Cohen’s Kappa.

Results: Feedback was collected from two emergency departments. Refinement cycles resulted in a dashboard that allows decision-makers to compare practice at a nursing level while offering near-real-time feedback and access to patient visit-level data. Feedback resulted in the addition of Systemic Inflammatory Response Syndrome (SIRS) related errors to the model. There were 10,000 patient visits considered for analysis. Patients less than 18 years of age (3%, n=306 visits) and visits with incomplete data (n=86) were removed. The final sample was 9608 visits.

Four categories of error were assessed: High-acuity, low-acuity, pain, and SIRS-criteria. Automated screening of high and low acuity errors had 100% sensitivity, specificity, and accuracy (κ =1, p<0.001) and outperformed expert review for high-risk (sensitive=97.8%, specificity=100%, accuracy=99.98%; κ =0.989, p,0.001) and low-risk errors (sensitivity=98.9%, specificity=100%, accuracy=99.98%; κ =0.994, p,0.001) compared to gold-standard expert manual review. Computer screening outperformed (sensitivity=71.3%, specificity=96.7%, accuracy=96.27%; κ =0.39, p,0.001) compared to expert assessment for pain errors (sensitivity=52.0%, specificity=97.4%, accuracy=96.58%; κ =0.335, p<0.001). Expert screening (sensitivity=100%, specificity=99.9%, accuracy=99.9%; κ =0.962, p<0.001); outperformed computer screening (sensitivity=7.9%, specificity=100%, accuracy=98.91%; κ =0.145, p<0.001) for SIRS-related errors.    

Advice and Lessons Learned:

1. Maintaining data integrity for the dashboard required multiple rounds of feedback and clinician input.
2. Because of the small sample and subsequent exclusion of pediatric patients the current dashboard and error detections thresholds levels for pediatric vital sign modifiers will need further validation before being used for pediatric ED patients.
3. Further refinements to the tool are needed to incorporate previously used text-parsing algorithms into the detection of pain-related triage errors.