Clinical trial datasets are usually collated and analysed in a single environment, allowing for flexible analysis including adjustment or stratification by subgroup. However, external factors are culminating in data being held in different environments, which could affect future design and analyses of trials.

Governance of sensitive data has evolved in response to digital and technological advances, with increasing emphasis on data security and protection, especially in health research. The Five Safes framework (safe data, projects, people, settings, and outputs) pioneered by the UK Office for National Statistics is widely adopted by data controllers in secure data environments (SDEs), also known as trusted research environments or data safe havens [1–4]. Following the Life Sciences Vision (2021) [5, 6] and Goldacre Review (2022) [7], there is now an accelerated move in the UK towards providing researchers access to health data safely via SDEs rather than distributing data to the researcher’s single location for a study. In SDEs, sets of defined data are securely provided by data controllers for appropriate analysis by validated ‘safe researchers’, which safeguard patient privacy and datasecurity.

Judicious use of high-quality data from health systems datasets (HSD) will streamline future clinical trials, facilitating planning, accrual and data collection [8–10]. Information about relevant individuals may be held across separate data locations, for example, multiple HSD of hospital admissions, primary care records and disease registries. Currently, clinical trial sponsors can collate the necessary HSD (‘bespoke extracts’) about their trial participants for detailed analysis of outcomes in their single environment [4, 11, 12]. In the near future in England, the default route to access these data is likely to be in SDEs, potentially based on geographical regions. SDEs operate with disclosure control processes in place, and individual level data (even if anonymised) often cannot easily egress an SDE. Therefore, a trial sponsor would likely be faced with the technical and governance challenges of performing HSD-based analyses in each environment separately and meta-analysing those results.

We sought to assess the impact on trial analyses if data are analysed in (or as if they are in) the same place. Where data can be pooled, the trial analysis becomes analogous to a ‘one-stage’ meta-analysis of individual participant data (IPD) [13], occasionally known as a ‘mega-analysis’ [14]. Any adjustment or stratification for clinical site or other grouping is straightforward and allows for a wide range of analysis possibilities. Where data cannot be pooled, we assumed isolated analyses would need to be done within each environment or silo, and the relevant summary data synthesised in a second analysis stage. This is analogous to a ‘two-stage’ meta-analysis approach, where the second stage is performed on aggregated data [15]. Hence, clinical trialists can learn from meta-analysts who are used to working across different clinical trials and, for various reasons, cannot always access IPD [16]. Meta-analysts usually prefer pooled IPD approaches over aggregated data for the depth and reliability of the analyses available, as well as simplicity [17].

We anticipated a two-stage meta-analytic approach may have an inflationary impact on the number of people required to participate in a study. Impact may be exacerbated by noncollapsibility, defined as “a failure of the measure when taken on a group to equal a simple average of the measure when taken on the group’s members or subgroups [18].” Theory suggests that siloing (splitting data) in noncollapsible settings would be more problematic (i.e. moving from a marginal estimand to one which conditions on silo) [19] and should lead to both larger magnitude of effect size and larger standard error, but similar significance levels[20].

The aims were to: explore any loss of precision if two-stage approaches were required; determine how to mitigate this upfront during sample size calculations; and elucidate longer term solutions.

We needed several different siloed datasets with varying outcomes. As datasets that were already siloed across separate SDEs were not readily available, we conducted simulated analyses in large, existing datasets from three clinical data sources. Silos were mimicked with datasets partitioned as if they had been split across siloed environments.

Each analysis of the three trials is shown below as a figure with tabulated summary information in the supplement, presented in outcome measure order. In each instance, the point estimate and standard error (SE) is shown for the reference analysis (intersection of the reference lines) and then overlaid for the one-stage (orange circle) and two-stage (purple circle) meta-analysis approaches. To simplify visualisation, results of 100 partitions (10% randomly selected from the 1,000 simulations) are displayed.

For the continuous outcome measure using linear regression in the breast cancer data, the one-stage analysis found the coefficient of hormonal therapy to be 180.6 days, with a standard error (SE) of 53.0 days. Figure 1 and Supplementary Table 1 show results of the one-stage analyses adjusted for silo and the two-stage analyses (purple circles), across different patterns of silos. For the two-stage analyses, the SEs generally increased while the estimates spread out as the number of silos increased.

Using the STAMPEDE prostate cancer dataset for the continuous outcome measure of log PSA value at 6 months, the estimate under the reference analysis was -0.25 and SE 0.07. The pattern of distribution of the estimates and SEs was similar to Figure 1 (Figure 2 and Supplementary Table 2).

For the binary outcome measure in the TB-IPD dataset, analysed with logistic regression, the reference analysis gave an estimated log OR of 0.45 with a SE of 0.10. As the number of silos increased for two-stage analysis, as well as the spread of the estimates and the level of the SEs increasing, the mean of the estimates also increased away from the null (see Figure 3 and Supplementary Table 3).

For the time-to-event analysis using the breast cancer data, the reference analysis gave a log hazard ratio of -0.38 with SE 0.13 (-0.36 and 0.13 respectively for the unadjusted analysis). As the silos were introduced and increased in number, the SEs increased (Figures 4 and 5 and Supplementary Tables 4 and 5); the same effect was observed as for the logistic regression. Changes in point estimates and standard errors were similar to those seen in Figure 3. However, in Figures 4 and 5 the positive shifts in point estimates represents a shift towards the null, whereas in Figure 3 the shift was away from the null. The spread of values is larger for the two-stage analyses compared to the corresponding one-stage ones with the impact most clearly observed with 25 silos.

For the STAMPEDE time-to-event analyses, the same pattern of analyses was followed for the time-to-event Cox regressions using breast cancer data. The reference adjusted analysis had its estimate as HR=0.78, with a standard error of 0.07. The mean point estimate does not change in Figure 6 and Supplementary Table 6, unlike Figure 3 where it goes away from the null and Figures 4, 5 and 7 (and Supplementary Table 7) where it goes towards the null.

Table 1 and Figure 8 present the indicative modifications to sample size that would have been required to address the observations in the previous examples. Accounting for silo in a one-stage approach, or in a two-stage approach with continuous or binary outcome, would generally require minimal change in sample size. However, a two-stage approach may require substantial inflation under Cox PH regression especially if, as is standard, adjusting for baseline characteristics. With ten silos, increases of 24% and 9% would have been required in the breast cancer and prostate cancer examples, and this was much higher with more silos. Each would require a longer trial period and/or faster accrual rates.

For all three outcome measure types, when more silos were used, standard errors increased while point estimates spread out, indicating loss of efficiency. The loss was greater if the participants were distributed across more silos and less evenly. This reduction of efficiency could be offset by inflating the target sample size.

We have shown that any analysis using a two-stage meta-analysis approach due to partitioning of data can be much less efficient compared to being able to do a one-stage approach where all the data contribute simultaneously to analytic models. The spread of estimated treatment effects is larger for the two-stage analyses compared to the corresponding one-stage analysis, with the impact most clearly observed with 25 silos. This problem is more pronounced for more complex outcome measures such as time-to-event(survival).

This loss of efficiency induced by a two-stage approach could be nominally offset by inflating the target sample size of the trial. The inflation factor was <10% for continuous or binary outcomes and for time-to-event outcomes with fewer than 9 silos, but reached 7-24% for time-to-event outcomes with 9 or 10 silos and 35-91% for time-to-event outcomes with 25 silos. We explored a variety of silo sizes, and we anticipated that small silos may, by chance, lead to more extreme estimates of the treatment effect. The nature of the problem was greater if the participants were split more widely and less evenly. This would have substantial implications for many clinical trials. Many trials pre-emptively inflate their sample size in anticipation of issues with follow-up [29]. The inflationary factor required because of a two-stage analytic approach would be separate to this and would be multiplicative rather than additive. For example, a phase III trial requiring 500 people (to observe 200 events) might be inflated by 10% to 550 to account for anticipated loss to follow-up and a further 20% to 660 to account for a two-stage analysis. Therefore, such required inflation in sample size may offset the streamlining gained by using HSD.

In binary outcome measures, two-stage analysis biased the estimated treatment effect away from the null (Figure 3), probably due to noncollapsibility of the odds ratio. In time-to-event data, two-stage analysis could instead bias the estimated treatment effect towards the null (Figures 4, 5 and 7), even though noncollapsibility of the hazard ratio would suggest a bias away from the null. We believe this bias arises from failure of the Normal approximation when the silo-level data become sparse (few events). [30] These biases need to be taken into account when comparing methods, but the relative biases seen in Figures 3–7 were of limited clinical importance, being between 10% away from the null and 8% towards the null, except with 25 silos in Figures 4, 5 and 7 where they were 13-23% towards the null (Supplementary figure 1).

Use of a two-stage meta-analysis approach also complicates the statistical analysis. Subgroup analyses, whether pre-specified or exploratory, may be impacted because data within a subgroup within a silo will be particularly sparse. Data separated by silos also raises challenges in forming appropriate imputation models to handle missing data [31].

If the data for participants are held in separate locations, some duplication of effort in information governance is likely, regardless of whether the data can be analysed together or only separately. For data held in multiple environments, resolution will be needed for archiving according to the relevant legislation and reconstructing datasets for secondary or tertiary data reuse projects. Better data federation is associated with closer alignment, harmonised governance and improved trust [32].

For all of these reasons, the conclusion reached for clinical trials is the same as that for meta-analysis: it is strongly preferable to interact with the data in a one-stage trial approach, where all the data are either actually in, or can be treated as if they are in, one place and included in the analysis model: preventing the problem is better than curingit [33].

The move towards ‘data access’ rather than ‘data sharing’ aims to restrict access to sensitive HSD [34]. SDEs are a safe and appropriate way to access information, particularly for projects where the participants have not given explicit consent, as in many retrospective real-world data projects [35]. For most clinical trials, there is an opportunity to obtain explicit consent which can include consent for the relevant, minimised data to be made safely available for analysis in the sponsor’s secure environment. Consistent forms of wording are required so data providers can egress datasets directly to trial sponsors for analysis [36].

One limitation of our work is that we did not access real HSD located in SDEs. Instead, entire clinical trial datasets were accessed and partitioned randomly to mimic having come from separate SDEs. However, this is also a strength: by creating many partitions of each dataset we were able to learn more about the broad pattern of what might happen, and to estimate a distribution of power loss. Previous authors have compared one-stage and two-stage approaches by simulating multiple trials [37], and by examining multiple different estimated quantities across the same four large trials [38]. However, their main objectives were to compare analysis approaches rather than to investigate the issue of siloing itself, and they did not examine time-to-event outcomes in which we have identified additional issues. Future research could explore a larger, more varied selection of real-world datasets, or a non-random partitioning mechanism.

The exact balance of participants across locations and SDEs for any particular trial may not be easily estimated up front. However, use of data in SDEs for trial planning, as well as delivery-through-data, should give a much better estimate of the actual numbers of participants that might be recruited and feed into sample size adjustment calculators.

The loss of efficiency due to siloing is not unique to clinical trials, but it is a particular issue for trials and other forms of prospective research predicated on a priori planning for a fixed number of participants to be recruited. For clinical trials with suitable participant consent, datasets could still be egressed from the relevant data providers, which may be multiple and shared in a safe and secure way to a safe and secure location from the trial sponsor, where they can interact with the data in one model. Alternatively, a full federated approach should be sought whereby the relevant, minimised datasets are pooled in one SDE where the trial sponsor’s representatives can interact with them in analysis. If pooling copies of datasets in one location is not possible, perhaps for reasons of information governance or file size, it will be preferable if technology were widely available to allow analysts to interact with the data as if they were in one place for one model, even though the datasets are actually in separate places. Some work is being started along these lines. Pilot studies run by TRE-FX and TELEPORT, supported by DARE UK, have developed approaches and technical solutions to federated analytics of genomic and health data [39, 40, 41]. Appropriate transformation of data assets into common data models may facilitate federated analytics; limitations in the mapping processes leading to data exclusion and loss of information have been reported [42, 43]. There are emerging technical approaches available for federated analytics although these have potential for statistical inefficiency. This work helps to demonstrate some benefits and risks of this mechanism and can help frame the utility of the technical innovation to support data that is held across multiple SDEs. There is still more to do to ensure full interaction with data, particularly with respect to clinical trials.

One response is for sponsors to continue with trial-specific data collection, an approach which is well understood, traditional, but duplicative of effort. Another is for development of new approaches to analysis that might mitigate the challenge of two-stage analysis across silos. A final response is for data providers to understand the implications of this work in order that they may sooner provide resolution towards one-stage analysis equivalent to pooling data in one place. In the NHS Research SDE Network, a network of geographically-defined SDEs with complete coverage in England and one national secure data environment, work towards full federation is ongoing. Multinational solutions are required. Many UK-wide trials will recruit from across regions covered by different SDEs; other countries have similar administrative divisions; and many clinical trials are multinational, particularly industry-led trials. There are many other blockers to using HSD for delivery-through-data of clinical trials. [8, 44] Trial sponsors may persist with tradition until all of these blockers have been addressed. Therefore, work is required rapidly to resolve each of these issues to make progress in streamlining the delivery of clinical trials.

There is a strong need to streamline clinical trials with better use of health systems data. However, there can be a penalty to be paid for any clinical trial that needs to use a two-stage aggregate data meta-analysis approach, which cannot easily be offset without increasing sample size as a minimum. Further work is required to enable one-stage analysis.

The salary of SBL was supported by UKRI (UK Research and Innovation; MRC (Medical Research Council) [grant number MC_UU_00004/08]. The salaries of IRW and DF were supported by UKRI [grant number MC_UU_00004/09]. The salaries of SF, SBL, MLM, MRS were supported by HDRUK: HDRUK2023.0025. MRS is employed by NHS England and contributed to this paper as part of the Data for Research and Development. This research was Powered by NHS Data.

We thank Andy Payne and Will Browne for critical comments during the drafting process.

We thank the investigators of, and all participants in, the STAMPEDE trial, German Breast Cancer study and each of the trials in the TB-IPD dataset for making data available for this project

This work uses data obtained through the WHO TB treatment individual patient data platform (TB-IPD), a collaborative initiative led by the World Health Organization (WHO), Global TB Programme and University College London (UCL).

Using CREDIT taxonomy