In many countries, governments routinely collect data on contacts with the healthcare system that includes patients’ sociodemographic information, diagnoses, procedures performed, and other parameters [1]. These data, referred to as administrative or claims data, can be used to address research questions related to epidemiology, clinical outcomes, and health economics. The advantages of using administrative data are that they are population-based, broadly available, and can be used to track outcomes of individuals longitudinally. In addition, accessing such data is relatively simpler, faster, and less expensive than conducting prospective studies [2–5].

Hospital administrative data is an appealing source for conducting broadly applicable neonatal research, as it contains many valuable maternal and neonatal variables related to preterm birth hospitalization. In Canada, this data is collected by the Canadian Institute of Health Information’s (CIHI) Discharge Abstract Database (DAD), which is available nationally. However, as always, the quality of research findings is dependent on the accuracy of research data, and there is a paucity of information on the validity of perinatal and neonatal data in the DAD. Only one prior Canadian study assessed the accuracy of DAD data for this purpose, and it included relatively few perinatal variables [6]. Also, there is a risk of biased conclusions in the above study due to the same data abstractors having collected both the administrative and reference data [6]. This lack of comprehensive validation of maternal and neonatal data abstracted by the DAD is a significant knowledge gap that needs to be addressed.

Hence our aim was to validate maternal and neonatal variables related to preterm birth in DAD against an independent reference standard. We hypothesized that the administrative data within DAD accurately represents these variables.

This study used data from the Winnipeg Health region in the Canadian province of Manitoba. Manitoba with a population of 1.39 million in 2020 has a universal, single payer healthcare system [7]. We used data held in the Population Research Data Repository at the Manitoba Centre for Health Policy (MCHP) [8]. The repository includes >100 diverse, de-identified datasets linked using encrypted personal health identification numbers.

All Canadian provinces and territories, except Quebec, collect administrative data from every hospitalisation in the standardised format of the DAD mandated by the CIHI [9]. Obtained in each hospital by centrally trained data abstractors, the DAD uses standardized definitions, data collection methods, and data entry software. DAD includes >200 data items, including patient demographics, timing of admission and disposition, up to 25 diagnoses delineated as pre-existing or hospital-acquired in ICD-10-CA format, and up to 15 procedures in Canadian Categorization of Interventions (CCI) format [10]. DAD collects information for all separations (discharge or death) from acute care institutions for all patients in Canada except for the province of Quebec. Maternal delivery details are collected in DAD and so are any hospital admissions mothers encounter during pregnancy. Diagnoses such as gestational diabetes or hypertension are entered by the medical provider in the mother’s chart during their admission that then gets abstracted in to the DAD by the data abstractor. All these hospital records are linked using a scrambled PHIN (Personal Health Identification Number) that is unique to each patient.

We assessed the accuracy of variables from the DAD (Supplementary Appendix 1) using the Canadian Neonatal Network (CNN) database as the reference standard [11]. CNN is a network of Level III neonatal intensive care units (NICUs) across 32 hospitals in Canada and one hospital in the U.S [11]. Data on maternal and neonatal variables for those admitted to participating NICUs are abstracted in a standardised way from hospital records by trained abstractors who were previous NICU nurses [12]. These data collectors are distinct from the DAD data abstractors who may not have prior NICU working experience. CNN has its own definitions for various diagnoses and interventions and do not rely on ICD-10-CA or CCI codes unlike DAD [12]. While the criteria for neonates entered into the CNN vary between NICUs, most NICUs including those in Winnipeg, collect data for all admitted neonates born <33 weeks gestational age. These data are compiled nationally, and being used for benchmarking, quality improvement and research purposes. The database has built-in checks at the data entry level as well as annually when data are cleaned before producing an annual report for the network, and if needed, data are rectified from source patient charts. Internal audit of CNN data has shown it to be highly accurate [13]. Both the DAD and CNN data are regularly imported to and available in MCHP.

All neonates born <33 weeks gestational age, born to mothers who were Manitoba residents, who were admitted to one of the two Winnipeg NICUs in the Winnipeg health region of Manitoba (2020 population 791,000) between January 1, 2010, and March 31, 2022, were identified from the DAD and formed the study population [7]. Of them, those whose DAD and CNN records were linked at MCHP were included in the study. The year 2010 was used as it was the first year that local CNN data were imported to MCHP. However, from 2018 onwards, the CNN data in Winnipeg was collected only for neonates <29 weeks gestation due to the limited resources available for data abstraction.

We assessed 55 parameters: 17 maternal variables, seven baseline neonatal variables, 22 neonatal diagnoses and outcomes, and nine procedures (Supplementary Appendix 1). Their definitions in the DAD are provided in Supplementary Appendix 2. Few parameters were not recorded either in the DAD or CNN for some neonates and hence were not included in the analysis; therefore, the total number of neonates varied for each parameter. Discharge information on re-admissions and hospital transfers are recorded in the DAD. However, the CNN usually captures only the birth admission; so, we compared only the birth admissions [12].

For categorical DAD parameters, we evaluated accuracy using sensitivity, specificity, and positive and negative predictive values, with 95% confidence intervals (CIs), against the reference standard. Sensitivity was defined as the proportion of true positives identified by DAD while specificity was the proportion of true negatives identified by DAD. Positive predictive value was defined as the proportion of positives identified by DAD that are true positives while negative predictive value was the proportion of negatives identified by DAD that are true negatives. We also calculated Cohen’s prevalence and bias corrected kappa coefficient between the categorical variables in the DAD and CNN with interpreted agreement being: <0.20 (poor), 0.20–0.39 (fair), 0.40–0.59 (moderate), 0.60–0.79 (good), and 0.80-1.00 (very good [14–16].

For continuous parameters, we evaluated accuracy between the DAD and CNN data as concordance, not correlation. For this purpose, we used Lin’s concordance correlation coefficient (LCCC) which is a measure of agreement between continuous variables categorized as: <0.90 (poor), 0.90-0.94 (moderate), 0.95-0.99 (substantial), and >0.99 (almost perfect) [17, 18]. We also produced Bland-Altman plots and limits of agreement for these parameters [18]. We conducted an a priori analysis of a few parameters stratified by when the NICUs transitioned to electronic medical records, i.e., birthdates ≤2018 versus >2018. We compared only sensitivity and specificity for these two periods using Fisher’s Exact test. P-values <0.05 were considered statistically significant.

This study was approved by the University of Manitoba Health Research Ethics Board and the Provincial Health Research Privacy Committee.

Of 9881 neonates born during the study period, 7797 were excluded (5871 had GA ≥33 weeks, 1926 were not admitted to NICU), leaving 2084 neonates born <33 weeks with data linked between DAD and CNN to be included in the study. Their characteristics, per CNN, are shown in Table 1. The majority of neonates were delivered by Caesarean section, while close to a quarter of them were multiples. A small proportion had maternal complications such as placental abruption, placenta previa, premature rupture of membranes, chorioamnionitis, and cervical insufficiency.

Validation of maternal and neonatal baseline variables are provided in Tables 2, 3 and Supplementary Appendix 4. The categorical maternal and neonatal parameters had a wide range of validity estimates. Some, such as substance use, had relatively low sensitivity, but high specificity and a fair agreement. Others, such as pre-gestational diabetes, had high sensitivity, low specificity and poor agreement. Few variables such as placenta previa, placental abruption (both fair correlation), multiples and neonatal variables such as small for gestational age and sex had relatively high sensitivity, specificity, and very good agreement. Maternal and neonatal baseline continuous variables had excellent agreement, except for the number of abortions and NICU length of stay, both of which had poor agreement. This is also shown in Supplementary Appendix 4, where gestational age, birth weight and date of birth the DAD data plotted close to the zero bias line while the data were more dispersed for the Apgar scores at 1 and 5 minutes.

Prevalence of neonatal outcomes are provided in Table 4. Validation results of the DAD neonatal outcomes and maternal and neonatal interventions are shown in Tables 4 and 5, respectively. These variables also showed a wide variability in their estimates, with some such as necrotizing enterocolitis with relatively high sensitivity and specificity and good agreement, while variables, such as sepsis and meningitis had relatively low sensitivity, high specificity and poor to fair agreement. Some of the neonatal procedures such as blood transfusion and patent ductus arteriosus (PDA) ligation had very good agreement.

Stratified analysis using the neonate’s year of birth (<2018 and >2018) did not show a significant difference between the two time periods except for sepsis, gestational diabetes, and pre-gestational diabetes. For sepsis, sensitivity improved slightly during the latter time period when patient records became electronic (Supplementary Appendix 3).

In this study, we sought to assess the accuracy of 55 parameters related to a preterm birth-related hospitalisation routinely collected in DAD over 12 years in two NICUs in Winnipeg, Canada. We found that a few baseline parameters, such as gestational age, birth weight, and maternal age had excellent agreement, while for others, the accuracy varied greatly across the baseline characteristics of the mother and child, neonatal outcomes, and procedures performed during the infant’s birth hospitalisation. As discussed above, there are important advantages of using administrative health data for neonatal research. For example, the DAD is the only database in Canada that collects hospitalisation data of all preterm deliveries (<37 weeks) at a national level. Those advantages are magnified by linking such data to other types of population-based data available in Canada. Extensive data repositories linking health and non-health datasets exist at the Manitoba Centre for Health Policy, the Institute for Clinical and Evaluative Sciences in Ontario, Population Data BC, and other sites [8, 19, 20]. For example, we linked the DAD with education data in Manitoba to assess the impact of preterm birth on school performance [21, 22] and linked it with outpatient health data to assess the association of preterm birth on subsequent maternal mental health [23]. Using administrative data for research therefore mandates that the data elements have acceptable accuracy. For research relating to prematurity, the most basic is the ability to accurately identify the gestational age, birthweight and sex of the infant. We found that the recording of those three fundamental parameters in the DAD are highly accurate. Also, the DAD is highly accurate in identifying birth by Caesarean section, multiple births, and Apgar scores. Overall, however, only 12 of the 55 parameters had accuracy (measured by kappa for binary variables and Lin’s concordance correlation coefficient for continuous ones) ≥0.90, while for 28 of them, the accuracy was <0.60. Accuracy was generally insufficient for research purposes for categorical maternal baseline variables, neonatal complications and procedures.

Only one prior study assessed the accuracy of maternal and neonatal administrative hospital data in Canada [6]. In comparison to ours, that study: included births at any gestational age instead of being restricted to preterm neonates as in our study, assessed only 15 parameters during the year 2002 only, compared the DAD to a reference standard only available in Nova Scotia, could have exaggerated the accuracy of variables by virtue of having the same data abstractors collect both the administrative and reference data, and considered gestational age as a binary variable with cut-off point at 37 weeks. Among the five parameters that the study had in common with ours, the identification of Caesarean delivery was likewise accurate. However, while the specificity of the DAD variables was generally high in both analyses, sensitivities differed. Our sensitivities were substantially lower for pre-gestational hypertension [30% (95% C.I. 20–41%) vs. 83% (74–91%)], respiratory distress syndrome (RDS) [82% (79–84%) vs. 94% (91–97%)] and severe intraventricular haemorrhage (IVH) [73% (66–80%) vs. 89% (52–100%)]. However, our sensitivity was slightly higher, though still low, for neonatal sepsis [46% (40–52%) vs. 39% (49–50%)]. Possible explanations for these differences include differential accuracy of the reference standards used; while both are believed to be accurate, no direct comparisons have been performed. Alternatively, the divergent findings could relate to differences in study cohorts or years of study. Less likely is that the DAD data collection systematically differs between locales, as abstraction personnel are uniformly trained and use uniform definitions, data collection methods, and data entry software.

One explanation for the poor validity of some of the parameters in DAD is likely due to the fact that DAD abstractors rely entirely on physician documentation to collect the data from patients’ medical records. As a result, they are likely to miss parameters that are not well documented by physicians. In contrast, in addition to physician notes, the CNN abstractors routinely access radiology and microbiology reports to confirm diagnoses such as sepsis or intraventricular haemorrhage leading to their better accuracy. A lack of significant improvement in the validity of DAD parameters after transitioning to electronic patient records highlights the potentially ongoing challenges with physician documentation.

Our study bridges an important knowledge gap by assessing the validity of preterm birth-related parameters in Canadian hospital abstracts that can be utilised for conducting research. It has notable strengths. Using a relatively large sample size, derived from 12 years of data, it assessed 55 maternal and neonatal parameters, using the CNN database as the reference, which is considered an accurate national standard for preterm outcomes [13]. Two limitations are that: (i) the performance estimates for outcomes with very low prevalence (e.g., meningitis) may be imprecise, as indicated by their very wide confidence intervals, and (ii) we did not include preterm neonates ≥33 weeks or parameters specific to them.

Our findings indicate that for research on children born preterm, Canadian hospital administrative data accurately identifies fundamental variables of gestational age, birthweight, Apgar scores, and multiple births. Beyond those however, most maternal baseline variables, neonatal outcomes and procedures are not accurately captured. These results highlight that for research using administrative data, investigators cannot assume the accuracy of key parameters in the DAD, but rather must locate or perform independent validation of them. Administrative data provides a valuable source of data that can be used to conduct population-based research. However, the accuracy of data is important to ensure the validity of research using this data. Our results showed that while some variables are accurate, others are not in DAD. It is important that researchers using this data are aware of these results and use only variables that are accurate. This study also highlights the need to improve the accuracy of data entered by the abstractors in DAD so that further training can be put in place to improve its accuracy. We recommend that CIHI takes steps, in collaboration with national organizations such as the Fetus and Newborn committee at the Canadian Pediatric Society (CPS) and Society of Obstetricians and Gynecologists of Canada (SOGC), to identify the source of these inaccuracies and to remedy them at the earliest possible time.