Validation of Preterm Birth Related Perinatal and Neonatal Data in the Canadian Discharge Abstract Database to Facilitate Long-term Outcomes Research of Individuals Born Preterm

Abstract

Introduction
The Canadian Institute of Health Information's (CIHI) Discharge Abstract Database (DAD) contains standardised administrative data on all hospitalisations in Canada, excluding Quebec.

Objectives
We aimed to validate preterm birth related perinatal and neonatal data in DAD by assessing its accuracy against the reference standard of the Canadian Neonatal Network (CNN) database.

Methods
We linked birth hospitalization data between the DAD and CNN databases for all neonates born $<$33 weeks gestational age (GA) admitted to the Neonatal Intensive Care Units in Winnipeg, Canada, between 2010 and 2022. A comprehensive list of maternal and neonatal variables relevant to preterm birth was chosen \textit{a priori} for validation. For categorical variables, we measured correlation using Cohen's weighted kappa (k) and for continuous variables, we measured agreement using Lin's concordance correlation coefficient (LCCC).

Results
2084 neonates were included (mean GA 29.4 ± 2.4 weeks; birth weight 1430 ± 461g). Baseline continuous maternal and neonatal variables showed excellent accuracy in DAD [Maternal age: LCCC = 0.99 (0.99, 0.99); GA: LCCC = 0.95 (0.95, 0.96); birth weight: LCCC = 0.97 (0.96, 0.97); sex: k = 0.99 (0.98-0.99)]. In contrast, the accuracy of the maternal baseline categorical variables and neonatal outcomes and interventions ranged from very good to poor [e.g., Caesarean section: k = 0.91 (0.89-0.93), pre-gestational diabetes: k = 0.04 (0.03-0.05), neonatal sepsis: k = 0.37 (0.31-0.42), bronchopulmonary dysplasia: k = 0.26 (0.19-0.33), neonatal laparotomy: k = 0.55 (0.43-067)].

Conclusion
Neonatal variables such as gestational age and birth weight had high accuracy in DAD, while the accuracy of maternal and neonatal morbidities and interventions were variable, with some being poor. Reasons for the inaccuracy of these variables should be identified and measures taken to improve them.

Introduction

In many countries, governments routinely collect data on contacts with the healthcare system that includes patients’ sociodemographic information, diagnoses, procedures performed, and other parameters [1]. These data, referred to as administrative or claims data, can be used to address research questions related to epidemiology, clinical outcomes, and health economics. The advantages of using administrative data are that they are population-based, broadly available, and can be used to track outcomes of individuals longitudinally. In addition, accessing such data is relatively simpler, faster, and less expensive than conducting prospective studies [2–5].

Hospital administrative data is an appealing source for conducting broadly applicable neonatal research, as it contains many valuable maternal and neonatal variables related to preterm birth hospitalization. In Canada, this data is collected by the Canadian Institute of Health Information’s (CIHI) Discharge Abstract Database (DAD), which is available nationally. However, as always, the quality of research findings is dependent on the accuracy of research data, and there is a paucity of information on the validity of perinatal and neonatal data in the DAD. Only one prior Canadian study assessed the accuracy of DAD data for this purpose, and it included relatively few perinatal variables [6]. Also, there is a risk of biased conclusions in the above study due to the same data abstractors having collected both the administrative and reference data [6]. This lack of comprehensive validation of maternal and neonatal data abstracted by the DAD is a significant knowledge gap that needs to be addressed.

Hence our aim was to validate maternal and neonatal variables related to preterm birth in DAD against an independent reference standard. We hypothesized that the administrative data within DAD accurately represents these variables.

Methods

This study used data from the Winnipeg Health region in the Canadian province of Manitoba. Manitoba with a population of 1.39 million in 2020 has a universal, single payer healthcare system [7]. We used data held in the Population Research Data Repository at the Manitoba Centre for Health Policy (MCHP) [8]. The repository includes >100 diverse, de-identified datasets linked using encrypted personal health identification numbers.

All Canadian provinces and territories, except Quebec, collect administrative data from every hospitalisation in the standardised format of the DAD mandated by the CIHI [9]. Obtained in each hospital by centrally trained data abstractors, the DAD uses standardized definitions, data collection methods, and data entry software. DAD includes >200 data items, including patient demographics, timing of admission and disposition, up to 25 diagnoses delineated as pre-existing or hospital-acquired in ICD-10-CA format, and up to 15 procedures in Canadian Categorization of Interventions (CCI) format [10]. DAD collects information for all separations (discharge or death) from acute care institutions for all patients in Canada except for the province of Quebec. Maternal delivery details are collected in DAD and so are any hospital admissions mothers encounter during pregnancy. Diagnoses such as gestational diabetes or hypertension are entered by the medical provider in the mother’s chart during their admission that then gets abstracted in to the DAD by the data abstractor. All these hospital records are linked using a scrambled PHIN (Personal Health Identification Number) that is unique to each patient.

We assessed the accuracy of variables from the DAD (Supplementary Appendix 1) using the Canadian Neonatal Network (CNN) database as the reference standard [11]. CNN is a network of Level III neonatal intensive care units (NICUs) across 32 hospitals in Canada and one hospital in the U.S [11]. Data on maternal and neonatal variables for those admitted to participating NICUs are abstracted in a standardised way from hospital records by trained abstractors who were previous NICU nurses [12]. These data collectors are distinct from the DAD data abstractors who may not have prior NICU working experience. CNN has its own definitions for various diagnoses and interventions and do not rely on ICD-10-CA or CCI codes unlike DAD [12]. While the criteria for neonates entered into the CNN vary between NICUs, most NICUs including those in Winnipeg, collect data for all admitted neonates born <33 weeks gestational age. These data are compiled nationally, and being used for benchmarking, quality improvement and research purposes. The database has built-in checks at the data entry level as well as annually when data are cleaned before producing an annual report for the network, and if needed, data are rectified from source patient charts. Internal audit of CNN data has shown it to be highly accurate [13]. Both the DAD and CNN data are regularly imported to and available in MCHP.

All neonates born <33 weeks gestational age, born to mothers who were Manitoba residents, who were admitted to one of the two Winnipeg NICUs in the Winnipeg health region of Manitoba (2020 population 791,000) between January 1, 2010, and March 31, 2022, were identified from the DAD and formed the study population [7]. Of them, those whose DAD and CNN records were linked at MCHP were included in the study. The year 2010 was used as it was the first year that local CNN data were imported to MCHP. However, from 2018 onwards, the CNN data in Winnipeg was collected only for neonates <29 weeks gestation due to the limited resources available for data abstraction.

We assessed 55 parameters: 17 maternal variables, seven baseline neonatal variables, 22 neonatal diagnoses and outcomes, and nine procedures (Supplementary Appendix 1). Their definitions in the DAD are provided in Supplementary Appendix 2. Few parameters were not recorded either in the DAD or CNN for some neonates and hence were not included in the analysis; therefore, the total number of neonates varied for each parameter. Discharge information on re-admissions and hospital transfers are recorded in the DAD. However, the CNN usually captures only the birth admission; so, we compared only the birth admissions [12].

For categorical DAD parameters, we evaluated accuracy using sensitivity, specificity, and positive and negative predictive values, with 95% confidence intervals (CIs), against the reference standard. Sensitivity was defined as the proportion of true positives identified by DAD while specificity was the proportion of true negatives identified by DAD. Positive predictive value was defined as the proportion of positives identified by DAD that are true positives while negative predictive value was the proportion of negatives identified by DAD that are true negatives. We also calculated Cohen’s prevalence and bias corrected kappa coefficient between the categorical variables in the DAD and CNN with interpreted agreement being: <0.20 (poor), 0.20–0.39 (fair), 0.40–0.59 (moderate), 0.60–0.79 (good), and 0.80-1.00 (very good [14–16].

For continuous parameters, we evaluated accuracy between the DAD and CNN data as concordance, not correlation. For this purpose, we used Lin’s concordance correlation coefficient (LCCC) which is a measure of agreement between continuous variables categorized as: <0.90 (poor), 0.90-0.94 (moderate), 0.95-0.99 (substantial), and >0.99 (almost perfect) [17, 18]. We also produced Bland-Altman plots and limits of agreement for these parameters [18]. We conducted an a priori analysis of a few parameters stratified by when the NICUs transitioned to electronic medical records, i.e., birthdates ≤2018 versus >2018. We compared only sensitivity and specificity for these two periods using Fisher’s Exact test. P-values <0.05 were considered statistically significant.

This study was approved by the University of Manitoba Health Research Ethics Board and the Provincial Health Research Privacy Committee.

Results

Of 9881 neonates born during the study period, 7797 were excluded (5871 had GA ≥33 weeks, 1926 were not admitted to NICU), leaving 2084 neonates born <33 weeks with data linked between DAD and CNN to be included in the study. Their characteristics, per CNN, are shown in Table 1. The majority of neonates were delivered by Caesarean section, while close to a quarter of them were multiples. A small proportion had maternal complications such as placental abruption, placenta previa, premature rupture of membranes, chorioamnionitis, and cervical insufficiency.

Table 1: Baseline characteristics of the study cohort in the Canadian Neonatal Network (CNN) database, N = 2084.

Maternal variables
Maternal age, mean±SD (Range)	29.8 ± 5.9 (14,46)
Gravidity, median [IQR]	2 [1,4]
Parity, median [IQR]	1 [0,2]
Smoking during pregnancy, N (%)	339 (16.6)
Substance use during pregnancy, N (%)	93 (4.5)
Pre-gestational hypertension, N (%)	76 (3.7)
Pre-gestational diabetes, N (%)	94 (4.6)
Gestational diabetes, N (%)	133 (6.5)
Caesarean section, N (%)	1219 (59.5)
Placental abruption, N (%)	17 (0.8)
Placenta previa, N (%)	10 (0.5)
Preterm premature rupture of membranes, N (%)	137 (6.7)
Cervical insufficiency, N (%)	13 (0.6)
Chorioamnionitis, N (%)	132 (6.4)
Multiple pregnancy, N (%)	466 (22.8)
Gestational hypertension, N (%)	240 (11.7)
Neonatal variables
Gestational age (weeks), mean±SD (Range)	29.4 ± 2.4 (21,32)
Birth weight (grams), mean±SD (Range)	1430 ± 461 (1010,3160)
Male, N (%)	1168 (57.0)
Apgar score at 1 min, median [IQR]	6 [4,8]
Apgar score at 5 min, median [IQR]	8 [6,9]
Small for gestational age, N (%)	175 (8.5)

Validation of maternal and neonatal baseline variables are provided in Tables 2, 3 and Supplementary Appendix 4. The categorical maternal and neonatal parameters had a wide range of validity estimates. Some, such as substance use, had relatively low sensitivity, but high specificity and a fair agreement. Others, such as pre-gestational diabetes, had high sensitivity, low specificity and poor agreement. Few variables such as placenta previa, placental abruption (both fair correlation), multiples and neonatal variables such as small for gestational age and sex had relatively high sensitivity, specificity, and very good agreement. Maternal and neonatal baseline continuous variables had excellent agreement, except for the number of abortions and NICU length of stay, both of which had poor agreement. This is also shown in Supplementary Appendix 4, where gestational age, birth weight and date of birth the DAD data plotted close to the zero bias line while the data were more dispersed for the Apgar scores at 1 and 5 minutes.

Table 2: Validation of the DAD maternal and neonatal categorical baseline variables. DAD, Discharge Abstract Database; PPV, Positive predictive value; NPV, Negative predictive value; PPROM, Preterm premature rupture of membranes.

	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Kappa coefficient (95% CI)
Maternal baseline variables
Smoking during pregnancy (N=1449)	0.75 (0.70,0.81)	0.92 (0.91,0.94)	0.68 (0.63,0.73)	0.94 (0.93,0.96)	0.65 (0.60,0.70)
Substance use during pregnancy (N=1449)	0.22 (0.13,0.31)	0.98 (0.97,0.99)	0.38 (0.25,0.51)	0.95 (0.94,0.96)	0.25 (0.15,0.35)
Pre-gestational hypertension (N=1368)	0.30 (0.20,0.41)	0.99 (0.99,1.00)	0.68 (0.52,0.84)	0.96 (0.95,0.97)	0.40 (0.28,0.51)
Gestational hypertension (N=1368)	0.40 (0.34,0.47)	0.98 (0.98,0.99)	0.83 (0.77,0.90)	0.89 (0.87,0.90)	0.49 (0.42,0.55)
Pre-gestational diabetes (N=1368)	1.00 (1.00,1.00)	0.23 (0.20,0.25)	0.09 (0.07,0.10)	1.00 (1.00,1.00)	0.04 (0.03,0.05)
Gestational diabetes (N=1368)	0.19 (0.12,0.26)	0.82 (0.80,0.84)	0.10 (0.06,0.14)	0.91 (0.89,0.92)	0.01 (0.00,0.06)
Placenta previa (N=626)	0.89 (0.68,1.00)	0.95 (0.93,0.97)	0.20 (0.07,0.33)	1.00 (1.00,1.00)	0.32 (0.15,0.49)
Placental abruption (N=626)	0.76 (0.56,0.97)	0.91 (0.89,0.93)	0.19 (0.10,0.28)	0.99 (0.99,1.00)	0.27 (0.15,0.39)
PPROM (N=786)	0.84 (0.78,0.91)	0.77 (0.74,0.80)	0.44 (0.38,0.50)	0.96 (0.94,0.98)	0.45 (0.38,0.52)
Cervical insufficiency (N=635)	0.85 (0.65,1.00)	0.96 (0.95,0.98)	0.32 (0.17,0.48)	1.00 (0.99,1.00)	0.45 (0.27,0.63)
Chorioamnionitis (N=635)	0.39 (0.24,0.55)	0.98 (0.97,0.99)	0.58 (0.39,0.77)	0.97 (0.95,0.98)	0.44 (0.29,0.60)
Multiples (N=2001)	0.95 (0.93,0.97)	1.00 (0.99,1.00)	0.99 (0.98,1.00)	0.99 (0.98,0.99)	0.96 (0.95,0.97)
Neonatal baseline variables
Sex (N=2078)	0.99 (0.99,1.00)	1.00 (0.99,1.00)	0.99 (0.99,1.00)	0.99 (0.99,1.00)	0.99 (0.98,0.99)
Small for gestational age (N=1417)	0.87 (0.81,0.93)	0.99 (0.98,0.99)	0.88 (0.82,0.94)	0.99 (0.98,0.99)	0.86 (0.81,0.91)

Table 3: Validation of the DAD maternal and neonatal continuous variables. DAD, Discharge Abstract Database; LCCC, Lin’s concordance correlation coefficient; NICU, neonatal intensive care unit.

	LCCC (95% CI)
Maternal baseline variables
Maternal age (N=1970)	0.99 (0.99,0.99)
Gravidity (N=1890)	0.97 (0.96,0.97)
Parity (N=1890)	0.97 (0.97,0.98)
Number of abortions (N=1722)	0.85 (0.83,0.86)
Neonatal baseline variables
Gestational age (N=1422)	0.95 (0.95, 0.96)
Birth weight (N=2077)	0.97 (0.96,0.97)
Birth date (N=2082)	1.00 (1.00, 1.00)
Apgar score at 1 min (N=1944)	0.98 (0.98,0.98)
Apgar score at 5 min (N=1946)	0.96 (0.95,0.96)
Neonatal outcome variable
NICU length of stay (N=2084)	0.88 (0.87,0.89)

Prevalence of neonatal outcomes are provided in Table 4. Validation results of the DAD neonatal outcomes and maternal and neonatal interventions are shown in Tables 4 and 5, respectively. These variables also showed a wide variability in their estimates, with some such as necrotizing enterocolitis with relatively high sensitivity and specificity and good agreement, while variables, such as sepsis and meningitis had relatively low sensitivity, high specificity and poor to fair agreement. Some of the neonatal procedures such as blood transfusion and patent ductus arteriosus (PDA) ligation had very good agreement.

Table 4: Validation of the DAD neonatal outcome categorical variables (N = 2084 unless otherwise specified). DAD, Discharge Abstract Database; PPV, Positive predictive value; NPV, Negative predictive value; s-suppressed due to confidentiality (N < 6).

	Prevalence N (%)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Kappa coefficient (95% CI)
Necrotizing enterocolitis	89 (4.3)	0.85 (0.78,0.93)	0.98 (0.98,0.99)	0.69 (0.60,0.78)	0.99 (0.99,1.00)	0.75 (0.68,0.82)
Gastrointestinal perforation	49 (2.4)	0.51 (0.37,0.65)	1.00 (0.99,1.00)	0.76 (0.61,0.90)	0.99 (0.98,0.99)	0.60 (0.48,0.73)
Mild intraventricular haemorrhage	310 (14.9)	0.62 (0.57,0.68)	0.91 (0.89,0.92)	0.54 (0.49,0.59)	0.93 (0.92,0.94)	0.50 (0.45,0.55)
Severe intraventricular haemorrhage	148 (7.1)	0.73 (0.66,0.80)	0.87 (0.68,0.89)	0.30 (0.26,0.35)	0.98 (0.97,0.98)	0.36 (0.31,0.42)
Any intraventricular haemorrhage	458 (22.0)	0.66 (0.61,0.70)	0.97 (0.96,0.98)	0.85 (0.81,0.88)	0.91 (0.90,0.92)	0.68 (0.64,0.72)
Periventricular leukomalacia	46 (2.2)	0.50 (0.36,0.64)	1.00 (1.00,1.00)	0.96 (0.88,1.00)	0.99 (0.98,0.99)	0.65 (0.52,0.78)
Sepsis	261 (12.5)	0.46 (0.40,0.52)	0.91 (0.90,0.93)	0.43 (0.38,0.49)	0.92 (0.91,0.93)	0.37 (0.31,0.42)
Meningitis	S	0.50 (0.01,0.99)	0.99 (0.99,0.99)	0.09 (0.00,0.20)	1.00 (1.00,1.00)	0.15 (0.0,0.33)
Patent ductus arteriosus	432 (20.7)	0.81 (0.77,0.85)	0.98 (0.98,0.99)	0.93 (0.91,0.96)	0.95 (0.94,0.96)	0.83 (0.80,0.86)
Retinopathy of prematurity	383 (18.4)	0.61 (0.56,0.66)	0.96 (0.95,0.97)	0.79 (0.74,0.83)	0.92 (0.90,0.93)	0.63 (0.59,0.68)
Bronchopulmonary dysplasia	93 (4.5)	0.48 (0.38,0.59)	0.92 (0.91,0.93)	0.22 (0.16,0.28)	0.97 (0.97,0.98)	0.26 (0.19,0.33)
Seizures	32 (1.5)	0.41 (0.24,0.58)	1.00 (1.00,1.00)	0.72 (0.52,0.93)	0.99 (0.99,0.99)	0.51 (0.35,0.68)
Death (N = 1881)	133 (7.1)	0.70 (0.62,0.78)	1.00 (0.99,1.00)	0.97 (0.93,1.00)	0.97 (0.97,0.98)	0.80 (0.74,0.86)
Pneumothorax	102 (4.9)	0.78 (0.70,0.86)	1.00 (1.00,1.00)	0.98 (0.94,1.00)	0.99 (0.98,0.99)	0.86 (0.81,0.92)
Respiratory distress syndrome	1075 (51.6)	0.82 (0.79,0.84)	0.54 (0.50,0.56)	0.65 (0.63,0.68)	0.73 (0.70,0.76)	0.35 (0.32,0.39)
Neonatal thrombosis (N = 1368)	23 (1.7)	0.39 (0.19,0.59)	0.99 (0.99,1.00)	0.56 (0.32,0.81)	0.99 (0.98,1.00)	0.45 (0.26,0.65)
Spontaneous intestinal perforation	23 (1.1)	0.74 (0.56,0.92)	0.99 (0.99,1.00)	0.50 (0.33,0.67)	1.00 (0.99,1.00)	0.59 (0.44,0.74)
Delayed cord clamping (N = 1198)	936 (78.1)	0.74 (0.56,0.92)	0.99 (0.99,1.00)	0.50 (0.33,0.67)	1.00 (0.99,1.00)	0.59 (0.44,0.74)
Chest compression	50 (2.4)	0.50 (0.36,0.64)	0.99 (0.99,1.00)	0.63 (0.48,0.78)	0.99 (0.98,0.99)	0.55 (0.42,0.67)
Congenital anomalies	230 (11.0)	0.85 (0.81,0.90)	0.74 (0.72,0.76)	0.29 (0.26,0.33)	0.98 (0.97,0.98)	0.32 (0.28,0.36)
Urinary tract infection	27 (1.3)	0.85 (0.72,0.99)	1.00 (0.99,1.00)	0.74 (0.59,0.90)	1.00 (1.00,1.00)	0.79 (0.67,0.91)
Resuscitation medications	24 (1.2)	0.42 (0.22,0.61)	0.90 (0.89,0.92)	0.05 (0.02,0.08)	0.99 (0.99,1.00)	0.02 (0.02,0.11)

Table 5: Validation of the DAD maternal and neonatal procedures (N = 2084 unless otherwise specified). PPV, Positive predictive value; NPV, Negative predictive value; s-suppressed due to confidentiality (N<6).

	Prevalence N (%)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Kappa coefficient (95% CI)
Maternal procedure
Caesarean section	1219 (58.5)	0.98 (0.97,0.99)	0.93 (0.91,0.94)	0.95 (0.94,0.96)	0.97 (0.96.0.98)	0.91 (0.89,0.93)
Neonatal procedures
Blood transfusion	557 (26.7)	0.88 (0.85,0.91)	0.98 (0.98,0.99)	0.95 (0.93,0.97)	0.96 (0.95,0.97)	0.88 (0.86,0.91)
Mechanical ventilation (N = 1449)	517 (35.7)	0.91 (0.89,0.94)	0.47 (0.44,0.51)	0.49 (0.46,0.52)	0.91 (0.88, 0.94)	0.33 (0.29,0.36)
Ventriculoperitoneal shunt insertion	8 (0.4)	1.00 (1.00,1.00)	0.99 (0.99,1.00)	0.42 (0.20,0.64)	1.00 (1.00,1.00)	0.59 (0.37,0.81)
Tracheostomy	S	1.00 (1.00,1.00)	1.00 (1.00,1.00)	0.25 (0.00,0.67)	1.00 (1.00,1.00)	0.40 (0.00,0.94)
Gastrostomy (N = 958)	S	0.67 (0.29,1.00)	0.99 (0.98,0.99)	0.25 (0.04,0.46)	1.00 (0.99,1.00)	0.36 (0.10,0.62)
Laparotomy	52 (2.5)	0.52 (0.38,0.66)	0.99 (0.99,1.00)	0.61 (0.47,0.76)	0.99 (0.98,0.99)	0.55 (0.43,0.67)
Patent ductus arteriosus ligation	38 (1.8)	0.89 (0.80,0.99)	1.00 (1.00,1.00)	0.92 (0.83,1.00)	1.00 (1.00,1.00)	0.91 (0.84,0.97)
Peritoneal drainage (N = 1198)	22 (1.8)	0.50 (0.29,0.71)	1.00 (1.00,1.00)	0.85 (0.65,1.00)	0.99 (0.99,1.00)	0.62 (0.43,0.81)

Stratified analysis using the neonate’s year of birth (<2018 and >2018) did not show a significant difference between the two time periods except for sepsis, gestational diabetes, and pre-gestational diabetes. For sepsis, sensitivity improved slightly during the latter time period when patient records became electronic (Supplementary Appendix 3).

Discussion

In this study, we sought to assess the accuracy of 55 parameters related to a preterm birth-related hospitalisation routinely collected in DAD over 12 years in two NICUs in Winnipeg, Canada. We found that a few baseline parameters, such as gestational age, birth weight, and maternal age had excellent agreement, while for others, the accuracy varied greatly across the baseline characteristics of the mother and child, neonatal outcomes, and procedures performed during the infant’s birth hospitalisation. As discussed above, there are important advantages of using administrative health data for neonatal research. For example, the DAD is the only database in Canada that collects hospitalisation data of all preterm deliveries (<37 weeks) at a national level. Those advantages are magnified by linking such data to other types of population-based data available in Canada. Extensive data repositories linking health and non-health datasets exist at the Manitoba Centre for Health Policy, the Institute for Clinical and Evaluative Sciences in Ontario, Population Data BC, and other sites [8, 19, 20]. For example, we linked the DAD with education data in Manitoba to assess the impact of preterm birth on school performance [21, 22] and linked it with outpatient health data to assess the association of preterm birth on subsequent maternal mental health [23]. Using administrative data for research therefore mandates that the data elements have acceptable accuracy. For research relating to prematurity, the most basic is the ability to accurately identify the gestational age, birthweight and sex of the infant. We found that the recording of those three fundamental parameters in the DAD are highly accurate. Also, the DAD is highly accurate in identifying birth by Caesarean section, multiple births, and Apgar scores. Overall, however, only 12 of the 55 parameters had accuracy (measured by kappa for binary variables and Lin’s concordance correlation coefficient for continuous ones) ≥0.90, while for 28 of them, the accuracy was <0.60. Accuracy was generally insufficient for research purposes for categorical maternal baseline variables, neonatal complications and procedures.

Only one prior study assessed the accuracy of maternal and neonatal administrative hospital data in Canada [6]. In comparison to ours, that study: included births at any gestational age instead of being restricted to preterm neonates as in our study, assessed only 15 parameters during the year 2002 only, compared the DAD to a reference standard only available in Nova Scotia, could have exaggerated the accuracy of variables by virtue of having the same data abstractors collect both the administrative and reference data, and considered gestational age as a binary variable with cut-off point at 37 weeks. Among the five parameters that the study had in common with ours, the identification of Caesarean delivery was likewise accurate. However, while the specificity of the DAD variables was generally high in both analyses, sensitivities differed. Our sensitivities were substantially lower for pre-gestational hypertension [30% (95% C.I. 20–41%) vs. 83% (74–91%)], respiratory distress syndrome (RDS) [82% (79–84%) vs. 94% (91–97%)] and severe intraventricular haemorrhage (IVH) [73% (66–80%) vs. 89% (52–100%)]. However, our sensitivity was slightly higher, though still low, for neonatal sepsis [46% (40–52%) vs. 39% (49–50%)]. Possible explanations for these differences include differential accuracy of the reference standards used; while both are believed to be accurate, no direct comparisons have been performed. Alternatively, the divergent findings could relate to differences in study cohorts or years of study. Less likely is that the DAD data collection systematically differs between locales, as abstraction personnel are uniformly trained and use uniform definitions, data collection methods, and data entry software.

One explanation for the poor validity of some of the parameters in DAD is likely due to the fact that DAD abstractors rely entirely on physician documentation to collect the data from patients’ medical records. As a result, they are likely to miss parameters that are not well documented by physicians. In contrast, in addition to physician notes, the CNN abstractors routinely access radiology and microbiology reports to confirm diagnoses such as sepsis or intraventricular haemorrhage leading to their better accuracy. A lack of significant improvement in the validity of DAD parameters after transitioning to electronic patient records highlights the potentially ongoing challenges with physician documentation.

Our study bridges an important knowledge gap by assessing the validity of preterm birth-related parameters in Canadian hospital abstracts that can be utilised for conducting research. It has notable strengths. Using a relatively large sample size, derived from 12 years of data, it assessed 55 maternal and neonatal parameters, using the CNN database as the reference, which is considered an accurate national standard for preterm outcomes [13]. Two limitations are that: (i) the performance estimates for outcomes with very low prevalence (e.g., meningitis) may be imprecise, as indicated by their very wide confidence intervals, and (ii) we did not include preterm neonates ≥33 weeks or parameters specific to them.

Conclusion

Our findings indicate that for research on children born preterm, Canadian hospital administrative data accurately identifies fundamental variables of gestational age, birthweight, Apgar scores, and multiple births. Beyond those however, most maternal baseline variables, neonatal outcomes and procedures are not accurately captured. These results highlight that for research using administrative data, investigators cannot assume the accuracy of key parameters in the DAD, but rather must locate or perform independent validation of them. Administrative data provides a valuable source of data that can be used to conduct population-based research. However, the accuracy of data is important to ensure the validity of research using this data. Our results showed that while some variables are accurate, others are not in DAD. It is important that researchers using this data are aware of these results and use only variables that are accurate. This study also highlights the need to improve the accuracy of data entered by the abstractors in DAD so that further training can be put in place to improve its accuracy. We recommend that CIHI takes steps, in collaboration with national organizations such as the Fetus and Newborn committee at the Canadian Pediatric Society (CPS) and Society of Obstetricians and Gynecologists of Canada (SOGC), to identify the source of these inaccuracies and to remedy them at the earliest possible time.

Acknowledgements

The authors acknowledge the Manitoba Centre for Health Policy for the use of data contained in the Manitoba Population Research Data Repository under project # HS25630 (H2022:258) (HIPC#2019/2020-43). The results and conclusions are those of the authors, and no official endorsement by the Manitoba Centre for Health Policy, Manitoba Health, or other data providers is intended or should be inferred.

Funding support

This work was supported by the Thorlakson Foundation Fund at the University of Manitoba (UM Project # 57067) awarded to Dr. Deepak Louis in 2022

Conflicts of interest

The authors have no conflicts of interest to disclose.

Ethics statement

We obtained ethical approval from the University of Manitoba Bannatyne Research Ethics Board (REB). REB Registry Number: HS25630(H2022:258).

Data availability statement

Data used in this article was derived from the Manitoba Population Research Data Repository under project # HS25630 (H2022:258) (HIPC#2019/2020-43), this is available at the Manitoba Centre for Health Policy (MCHP).

We obtained ethical approval from the University of Manitoba Bannatyne Research Ethics Board (REB). REB Registry Number: HS25630(H2022:258) to be able to access these records.

We do not own the original data and will not be able to provide it to a public repository. The data may be reviewed at MCHP with consent from the required privacy and ethical bodies.

References

1. Garland A, Gershengorn HB, Marrie RA, ines {Reider N, Wilcox ME. A Practical, Global Perspective on Using Administrative Data to Conduct Intensive Care Unit Research. Ann Am Thorac Soc. 2015 Sep;12(9):1373–86.

2. Modi N, Ashby D, Battersby C, Brocklehurst P, Chivers Z, Costeloe K, et al. Linking the National Neonatal Research Database to other NHS data sets; feasibility and birth cohort studies. In: Developing routinely recorded clinical data from electronic patient records as a national resource to improve neonatal health care: the Medicines for Neonates research programme [Internet]. NIHR Journals Library; 2019 [cited 2024 May 13]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK546526/.

3. Doiron D, Raina P, Fortier I, Linkage Between Cohorts and Health Care Utilization Data: Meeting of Canadian Stakeholders workshop participants. Linking Canadian Population Health Data: Maximizing the Potential of Cohort and Administrative Data. Can J Public Health [Internet]. 2013 May 1 [cited 2024 May 10];104(3):e258–61. Available from: 10.17269/cjph.104.3775.

   10.17269/cjph.104.3775

4. Maret-Ouda J, Tao W, Wahlin K, Lagergren J. Nordic registry-based cohort studies: Possibilities and pitfalls when combining Nordic registry data. Scand J Public Health [Internet]. 2017 Jul 1 [cited 2024 May 10];45(17_suppl):14–9. Available from: 10.1177/1403494817702336.

   10.1177/1403494817702336

5. Thygesen LC, Ersbøll AK. When the entire population is the sample: strengths and limitations in register-based epidemiology. Eur J Epidemiol [Internet]. 2014 Aug 1 [cited 2024 May 10];29(8):551–8. Available from: 10.1007/s10654-013-9873-0.

   10.1007/s10654-013-9873-0

6. Joseph KS, Fahey J, Canadian Perinatal Surveillance System. Validation of perinatal data in the Discharge Abstract Database of the Canadian Institute for Health Information. Chronic Dis Can. 2009;29(3):96–100.

7. Population Report | Health [Internet]. Province of Manitoba - Health. [cited 2024 May 10]. Available from: https://www.gov.mb.ca/health/population/index.html.

8. Katz A, Enns J, Smith M, Burchill C, Turner K, Towns D. Population Data Centre Profile: The Manitoba Centre for Health Policy. Int J Popul Data Sci [Internet]. 2020 Feb 25 [cited 2024 May 10];4(2). Available from: https://ijpds.org/article/view/1131.

9. Discharge Abstract Database (DAD) metadata | CIHI [Internet]. [cited 2024 May 10]. Available from: https://www.cihi.ca/en/discharge-abstract-database-dad-metadata.

10. Canadian Coding Standards for Version 2022 ICD-10-CA and CCI. [Internet]. [cited 2024 May 10]. Available from: https://secure.cihi.ca/free_products/canadian-coding-standards-2022-en.pdf.

11. The Canadian Neonatal NetworkTM [Internet]. [cited 2024 May 10]. Available from: https://www.canadianneonatalnetwork.org/portal/.

12. CNN Manual_20230308.pdf [Internet]. [cited 2024 May 11]. Available from: https://www.canadianneonatalnetwork.org/portal/Portals/0/CNN%20Manuals/CNN%20Manual_20230308.pdf.

13. Shah PS, Seidlitz W, Chan P, Yeh S, Musrap N, Lee SK, et al. Internal Audit of the Canadian Neonatal Network Data Collection System. Am J Perinatol. 2017 Oct;34(12):1241–9.

14. Sim J, Wright CC. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Phys Ther [Internet]. 2005 Mar 1 [cited 2024 May 10];85(3):257–68. Available from: 10.1093/ptj/85.3.257.

    10.1093/ptj/85.3.257

15. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–74.

16. Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: Measures of agreement. Perspect Clin Res. 2017;8(4):187–91.

17. Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989 Mar;45(1):255–68.

18. Mahon G. A Proposal for Strength-of-Agreement Criteria for Lin’s Concordance Correlation Coefficient. In 2005 [cited 2024 May 10]. Available from: https://www.semanticscholar.org/paper/A-Proposal-for-Strength-of-Agreement-Criteria-for-Mahon/0bed3bdb75247d09d4c5f1398cd309272fd26ad5.

19. ICES | Use ICES Data | Data Dictionary [Internet]. ICES. [cited 2024 May 10]. Available from: https://www.ices.on.ca/use-ices-data/data-dictionary/.

20. Data available | Population Data BC [Internet]. [cited 2024 May 10]. Available from: https://www.popdata.bc.ca/data.

21. School Readiness Among Children Born Preterm in Manitoba, Canada | Pediatrics | JAMA Pediatrics | JAMA Network [Internet]. [cited 2024 May 10]. Available from: https://jamanetwork.com/journals/jamapediatrics/fullarticle/2795002.

22. Louis D, Oberoi S, Ricci FM, Pylypjuk C, Alvaro R, Seshia M, et al. Grade 3 school performance among children born preterm: a population-based cohort study. Arch Dis Child Fetal Neonatal Ed. 2023 May;108(3):286–93.

23. Louis D, Akil H, Bolton JM, Bacchini F, Netzel K, Oberoi S, et al. Mental Disorders Among Mothers of Children Born Preterm: A Population-Based Cohort Study in Canada. Can J Psychiatry Rev Can Psychiatr. 2023 May;68(5):338–46.