Enhanced Depression Phenotyping Using Machine Learning and Electronic Medical Records for Disease Surveillance: A Validation Study in Inpatient Settings

Abstract

Background
Accurate detection of inpatient depression is essential for surveillance and resource allocation. Traditional International Classification of Diseases (ICD) codes suffer from low sensitivity and delayed availability.

Objective
To develop and validate machine learning (ML) phenotyping algorithms for depression using unstructured inpatient clinical notes from electronic medical records (EMRs), compared with ICD-10-CA codes and manual chart review.

Methods
This retrospective cohort study included a derivation cohort (N=3032 adults, Jan–June 2015) and a temporally independent external validation cohort (N=10659 adults, 2017–2022) from Calgary acute care hospitals. Both cohorts were deterministically linked to DAD and EMR databases. Manual chart reviews defined the reference standard for depression status. We compared ICD-10-CA codes against three EMR-based algorithms: Keyword Search, Concept Model and Document-Concept Model. The latter two employed a supervised ML classifier (XGBoost) utilizing clinical concepts extracted via the cTAKES natural language processing (NLP) pipeline. Primary measures were prevalence, sensitivity, positive predictive value (PPV), specificity, negative predictive value (NPV), and F1 score.

Results
In derivation cohort (prevalence: 18.1%), the Concept Model achieved 76.4% sensitivity and 76.7% F1 score, significantly outperforming Keyword Search (F1: 66.7%), Document-Concept Model (F1: 73.7%), and ICD-10-CA codes (sensitivity: 20.0%; F1: 32.6%). In external validation (prevalence: 13.1%), the Concept Model retained robust performance (sensitivity: 92.3%; F1: 80.1%), compared with ICD-10-CA (sensitivity: 6.1%; F1: 11.4%).

Conclusion
An EMR-based Concept Model using NLP and ML markedly improves depression case detection over ICD coding in inpatient settings. The methods could potentially facilitate psychiatric surveillance and epidemiological research using real-world clinical data.