IJPDS International Journal of Population Data Science IJPDS 2399-4908 Swansea University 10.23889/ijpds.v11i5.3674 11:5:3674 S2399490821036740 Population Data Science Accuracy of EMR-based algorithms for Adverse Event Detection SouthernDanielle1 CheligeerCheligeer2 WuGuosong3 EastwoodCathy1 ZengYong4 BoussatBastien5 FlemonsWard6 ForsterAlan7 XuYuan8 QuanHude9 Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, Canada Alberta Health Services, Calgary, Canada Department of Healthcare Analytics, Cape Breton University, Sydney, Canada Concordia Institute for Information Systems Engineering, Gina Cody School of Engineering and Computer Science, Concordia University, Montreal, Canada Clinical Epidemiology and Quality of Care Unit, University Grenoble Alpes, Faculty of Medicine, Grenoble University Hospital, Grenoble, France Department of Medicine, Cumming School of Medicine, University of Calgary, Calgary, Canada Department of Medicine, Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada Department of Oncology, Cumming School of Medicine, University of Calgary, Calgary, Canada; Department of Surgery, Cumming School of Medicine, University of Calgary, Calgary, Canada Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada; Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, Canada 11 5 3674 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. This article is available from the IJPDS website at: https://ijpds.org/article/view/3674 Background

Adverse events (AEs) are often measured using administrative data, which has validity limitations. Electronic Medical Records (EMRs) provide rich, unstructured data that can improve AE detection through data science methods. Objective: To evaluate the accuracy of EMR-based algorithms for AE detection and estimate AE prevalence.

 Methods

We conducted a retrospective chart review validation study at a tertiary-care hospital (population ∼1.5M). Adult patients admitted between 2017–2022 with discharge summaries or narrative EMRs were included; only the most recent admission was analyzed. Seventeen algorithms were developed for specific AE categories and an overall “any AE” category using natural language processing (NLP) and machine learning. Prevalence, sensitivity, specificity, PPV, NPV, and model performance (AUC, accuracy, F1 score) were calculated using chart review as a reference.

 Results

Of 10,939 charts, 10,659 were linked to discharge data; 2,069 patients had ≥1 AE. AE prevalence ranged from 0.2%–6.0%. Sensitivity and specificity varied (57%–100%, 74%–96%). For “any AE,” sensitivity was 63%, specificity 94%, PPV 72%, NPV 91%. Overall algorithm performance: AUC 78.4% (95% CI, 76.0–80.8), accuracy 87.6%, F1 score 67.2.

 Conclusion

NLP-based algorithms applied to EMRs show strong performance despite low AE prevalence, offering a scalable approach to enhance AE surveillance and patient safety.