Evidence for successful early years interventions is lacking, despite a renewed policy interest in providing all children with the best start in life [1, 3]. Despite its limitations, utilising RCD as an outcome could overcome several evaluation challenges. Hence, a systematic understanding of what components of RCD are being, or have been, used to evaluate early years interventions and/or policies, is crucial for planning of future evaluations. This may enable identification of areas where improvements to existing RCD infrastructures may be required, and identification of RCD that is being underused. Given the geographical variability of what and how data are collected between countries, it is important to consider these questions within the specific area of interest, in this case the United Kingdom.

This scoping review therefore aims to gain an overview of both planned and completed studies that have used an outcome measure obtained from RCD in evaluations of early years interventions (delivered either during pregnancy, or up child age 5 years old). Our focus is on universal and targeted interventions with a social and behavioural component, rather than interventions or products to treat specific conditions. The scoping review question is:

The ‘Population, Concept, Context’ framework was used to describe the eligible studies. Broadly, we focused on studies that included interventions delivered to caregivers with a 0-5-year-old child, and/or interventions delivered directly to children aged 0-5 years. The concept of the interventions was broad and could be any kind of intervention and/or policy that was delivered when parents were pregnant, or when the child was aged 0-5 years old. The context was UK only, as the purpose was to inform the use of routine data use in intervention evaluations within UK contexts. Studies published since 1st January 2000 were included, to ensure that any results are relevant to modern data systems.

Table 1 describes the framework and eligibility criteria. A more detailed table with exclusion criteria for guiding the screening process was published with the study protocol (see osf.io/cug36/). At abstract screening, we included any studies that: (a) were social and/or behavioural interventions and used a quantitative design, (b) were about parents/children (aged 0-5 years old) and (c) were in the UK, or it was not clear where the study took place. Clinical Trials of Investigational Medicinal Products (CTIMPs) were screened out at both abstract and full text screening stages, meaning most remaining intervention studies were social and/or behavioural. If any uncertainty arose regarding the eligibility of an intervention, two authors (KEM and DN) discussed and agreed on these. If both a study protocol and published results were available, we retained only the study results paper due to it containing more relevant information to our scoping review question. We hand searched for published studies resulting from protocols and instead included these where available, hence any protocol studies represent those not yet published.

Peer reviewed studies and grey literature (including reports, websites, and conference abstracts) were both eligible for inclusion. We included studies at any stage from study protocols, interim findings, to study reports of outcomes.

The full search is provided in Appendix 2. The words associated with the population, concept and context of the review were used to guide the search terms. The National Institute for Health and Care Excellence (NICE) filters were used to identify studies conducted within the United Kingdom [28]. Medline, Psycinfo and Embase were searched via Ovid for peer reviewed literature published between January 2000 and 13th November 2024.

Grey literature searches took place in November 2024. We targeted specific websites relevant to our topics, namely National Institute for Health Research (NIHR), Education Endowment Foundation (EEF), Early Intervention Foundation, National Foundation for Education Research (NFER), ISRCTN (https://www.isrctn.com/), Clinicaltrials.gov, GOV.UK(https://www.gov.uk/search/all), and GOV.wales (https://www.gov.wales/). Records of websites searched, number of potentially relevant items, and number scanned were kept in Microsoft Excel [29].

Titles and abstracts were uploaded to Covidence for screening. A piloting process took place with 100 studies, and agreement between all reviewers was reviewed. After piloting, amendments were made to the inclusion criteria to improve the clarity of the included studies. All titles and abstracts were screened by two reviewers according to the criteria described above for abstract screening. Cohen’s Kappa scores were reviewed regularly as a team, since lower than desired agreement (<0.60) occurred between some pairs of reviewers. Regular meetings and reviews took place with the whole team to resolve disagreements, and Cohen’s Kappas improved. After abstract screening, the full texts were screened and agreed by two reviewers (DN or KEM).

Data was extracted from eligible papers by three reviewers (RD, JG, KH). A proportion of 20% of selected articles was second checked by KEM to ensure accuracy. Data included specific details about the participants, concept, context, study methods and key detail regarding the outcome measurement, source of RCD, and data availability statements. Discussions sections from the included studies were examined by KEM to extract a list of specific strengths and limitations associated with using RCD that the study authors mentioned.

A full list of studies and characteristics is provided in Appendix 3. The included studies were summarised in terms of their study types, designs, intervention types, and outcomes used including the type of outcome, timing, and whether the outcome was reported at the parent or child level. Due to the primary interest of the study being in the outcome measures that were obtained from RCD, we extracted data on the specific outcome measures used by individual studies. KEM categorised these into broader outcome measures so that they could be summarised, and co-authors reviewed these. The resultant categories and examples of what specific measures they included were:

Data were charted by these study characteristics, and bar graphs created where they provided additional insights. Data was also extracted by KEM from individual studies’ strengths and limitations sections, and summarised according to broad themes, which were reviewed by co-authors.

Figure 1 presents the flow of included studies. There were two studies using the same data source, but they contained different periods of follow-up routine data, hence, we retained these two studies in the review. The studies are (1) a protocol of a randomised study to evaluate the Family Nurse Partnership [30] and the results from the routine data follow-up from that study [31].

Table 2 summarises key characteristics of the included studies, and the extracted data for all studies is provided in Appendix 3. This information is used in the narrative synthesis below.

We identified a total of 32 unique studies published between 2000-2024 that had used or were planning to use RCD as an outcome to evaluate an early years intervention. The majority were publications with results (76%), with the remainder being study protocols (24%). The studies were predominantly based in England (64%), followed by Scotland (24%), Wales (6%), or multiple nations (6%). The few studies using data from >1 nation may suggest a challenge in harmonising RCD across the UK’s devolved nations, since these nations operate separate early years services and hence separate data collection systems [16, 17]. The first eligible study was published in 2009. The number of eligible studies has increased over time, with the highest number being six in 2023. This could reflect both a developing field due to an increased policy demand for effective early years interventions [1, 65], and an increase in data linkage research and capabilities [13, 25].

Most studies employed observational designs (n = 23, 72%). Observational studies were likely most frequent due to the nature of RCD, which is that it is primarily collected for purposes other than research [66, 67]. RCTs typically represent funded studies which do not usually need to rely upon RCD. The most common type of intervention were perinatal programmes (29%). This may reflect the availability of RCD in maternity records, meaning these programmes are readily evaluable using RCD on birth related outcomes.

The common use of both parent and child outcomes demonstrates the benefits of using RCD, as it can assess the dual impact often targeted by complex interventions delivered in the early years (e.g., the Family Nurse Partnership, which aims to improve both maternal health/life chances and child outcomes [31, 41]). Evaluations that measure both categories are best positioned to capture the full scope of an intervention’s potential effectiveness.

Most studies measured more than one outcome type via RCD. This highlights a strength of using RCD, as it has the capacity for comprehensive and multi-purpose evaluation with no additional data collection burden for participants or researchers. This aggregation may allow evaluations to move beyond a single "primary" outcome, providing a more comprehensive understanding of an intervention’s effects across various domains. As noted by several of our included studies, use of RCD is highly cost-effective and low burden compared to collecting multiple different outcomes through bespoke surveys [42, 49].

The most common outcomes were birth outcomes, child education/development, and child health. The commonality of birth outcomes may, again, reflect the availability of RCD in maternity records to be used in such evaluations. These birth outcome data indicators may be recorded with higher completeness than other RCD, because they are embedded into midwifery practice, and essential for clinical decision-making. This is apparent in previous research which has assessed the potential utility of RCD by maternity services for research purposes, finding the completeness of this data to be acceptable [68]. These indicators therefore may serve as reliable, early indicators of intervention success, particularly for antenatal programs. The frequent use of the EYFSP to measure child development is likely due to it being a mandated assessment in England, meaning it has high coverage for this population at school entry [69]. Use of the EYFSP may allow researchers to assess the long-term developmental impact of perinatal interventions, a key advantage over short-term studies. This would also reflect a key policy interest, with the UK Government target for 75% of children to have a Good Level of Development on the EYFSP by 2028 [1].

The routinely collected ASQ®-3 by health visitors at the 2-2 1/2 year review completed in England’s Healthy Child Programme [16, 70] was indicated to be used by only one study, which was in Scotland [52]. The ASQ is a broad measure of a child’s early development and could be a targeted outcome for many interventions, and this suggests that this potential measure may be being underutilised, particularly in England. However, whilst the ASQ should be universally completed as a mandated assessment in England [16, 70], recent studies have indicated variation in the completeness of the ASQ [71], which may explain its underuse as an outcome in evaluations. All nations within the UK deliver universal services with routine assessments of early child development [72–74], however, nations appear to differ in when and how they measure child development, meaning it would be challenging to harmonise these data and evaluate the impact of UK wide policies on early child development. The two studies included in this review which included >1 UK nation focused on birth outcomes and child survival [55, 75], potentially reflecting this.

Whilst many studies targeted the same outcomes, they differed in their descriptions of their data source. For instance, several studies utilised routine maternity data, however, it was challenging to determine if they had used the same data sources e.g. "maternity electronic patient records," and "routinely collected NHS data". Studies also differed in whether they included data availability statements or not, with more recent studies being more likely to include them. Studies may have used the same underlying data source, or distinct datasets, and the variation in language and inconsistent use of data availability statements, particularly in older studies, makes this difficult to ascertain. The lack of standardisation in describing routine data sources presents a barrier for researchers seeking to use RCD to evaluate early years interventions.

The timing of the outcome varied, with the most common being up to 3 months post birth (32%), at birth (23%), and at child age 5 years (13%). This indicates that most studies used RCD in the very earliest years of children’s lives, and fewer studies successfully leveraged RCD for medium-to-long-term follow-up (with only 3 studies examining outcomes post 5 years old). A main benefit of RCD is the possibility for long-term follow-up with minimal participant burden. It is possible that as access to RCD improves, more studies linking long-term RCD to evaluations will emerge.

Most studies noted specific limitations in using RCD in their studies (64% of 33), with the most frequent being the limited choice of outcome measurement based on what was available in the RCD. This limitation highlights that crucial, but more nuanced, proximal outcomes (e.g., parental confidence, quality of parent-child interaction) that might typically be collected may be missed in a study relying upon RCD [19, 20, 41, 51]. This limitation will be compounded for quasi-experimental studies relying on RCD, as they will be unable to control for confounding by multiple risk factors which are not routinely measured – a limitation noted in one of our included studies [41]. Further, the measures collected in RCD may not be the most valid and reliable measures of specific outcomes. This could lead to an incomplete picture of an intervention’s effect, particularly if the mechanism of change is not directly measurable in RCD. Political interest in using population based data for evidence based decisions is crucial for developing new databases and data collection systems [7], hence the current Government Best Start policy [1] highlights the timeliness of this review.

Authors also noted concerns regarding accuracy of the data, and this should be investigated by researchers, for instance via studies which examine the implementation of RCD in local practice [76]. These limitations reflect that RCD are not designed for research purposes, and its use should always be considered with its limitations in mind [18, 21].

Fewer studies noted strengths related to using RCD (30% of 33 studies), which most frequently included an increased sample size and whole population coverage, and low missing data or low loss to follow-up. RCD may address previously noted challenges with evaluating early years interventions, including that such interventions are subject to a high attrition rate [8, 10]. Altogether, the limitations and strengths highlighted a critical trade-off when using RCD to evaluate interventions: whilst the use of RCD addresses some challenges with evaluating early years interventions, researchers are also limited by what was available in RCD and the quality of these data.

We did not formally assess the quality or risk of bias of the included studies. Consequently, we could not evaluate the quality of the RCD sources used or the potential impact of data quality on intervention evaluations. Since the use of RCD for research purposes is a rapidly evolving field, these review findings may require updating within a few years. Finally, although we made efforts to include grey literature, sources such as trial registrations and conference abstracts often lacked the detailed methodological information needed to confirm eligibility. This means the current review likely underestimates the volume of relevant studies that exist and may become more widely available in the near future.

Altogether, this review has highlighted a critical trade-off when using RCD to evaluate interventions: whilst RCD opens up evaluation opportunities, researchers are also limited by what measures are available in RCD. The recommendations call for (1) further research into which outcomes policymakers should integrate into RCD, (2) researchers to communicate findings using RCD to services; (3) researchers to clearly describe data access and sources, (4) researchers to investigate reasons for underrepresented outcomes, and (5) researchers to use both RCD and bespoke collected outcome to understand effects of interventions. With careful considerations, RCD can provide useful insights into the effectiveness of early years interventions, and researchers should continue to use it in combination with other available data. This scoping review reveals an emerging methodology that may have increasing importance for evaluating early years policies and interventions, but may currently be constrained by data availability and quality.