LML receives research funding from the Canadian Institutes of Health Research. The other authors have no conflicts of interest to declare.
Research and surveillance using routinely collected health data rely on algorithms or definitions to ascertain disease cases or health measures. Whenever algorithm validation studies are not possible due to the unavailability of a reference standard, algorithm feasibility studies can be used to create and assess algorithms for use in more than one population or jurisdiction. Publication of the methods used to conduct feasibility studies is critical for reproducibility and transparency. Existing guidelines applicable to feasibility studies include the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) and REporting of studies Conducted using Observational Routinely collected health Data (RECORD) guidelines. These guidelines may benefit from additional elements that capture aspects particular to multi-jurisdiction algorithm feasibility studies and ensure their reproducibility. The aim of this paper is to identify the minimum elements for reporting feasibility studies to ensure reproducibility and transparency.
A subcommittee of four individuals with expertise in routinely collected health data, multi-jurisdiction health research, and algorithm development and implementation was formed from Health Data Research Network (HDRN) Canada’s Algorithms and Harmonized Data Working Group (AHD-WG). The subcommittee reviewed items within the STROBE and RECORD guidelines and evaluated these items against published feasibility studies. Items to ensure transparent reporting of feasibility studies not contained within STROBE or RECORD guidelines were identified through consensus by subcommittee members using the Nominal Group Technique. The AHD-WG reviewed and approved these additional recommended elements.
Eleven new recommended elements were identified: one element for the title and abstract, one for the introduction, five for the methods, and four for the results sections. Recommended elements primarily addressed reporting jurisdictional data variabilities, data harmonization methods, and algorithm implementation techniques.
Implementation of these recommended elements, alongside the RECORD guidelines, is intended to encourage consistent publication of methods that support reproducibility, as well as increase comparability of algorithms and their use in national and international studies.
Research and surveillance studies that use routinely collected health data often rely on established algorithms (i.e., case or concept definitions) to identify diseases or conditions [
Transparent reporting of feasibility studies is an important facilitator for algorithm reuse and contributes to the FAIR (Findable, Accessible, Interoperable, and Reusable) principles of open data and open science [
Health Data Research Network (HDRN) Canada is a non-profit corporation founded in 2020 with the aim of facilitating the use of multi-jurisdictional electronic health and social data to drive improvements in health and health equity [
This paper includes a checklist of the recommended minimum elements for reporting algorithm feasibility studies to facilitate their reproducibility and transparency. These elements are meant to be used in addition to the STROBE and RECORD reporting guidelines.
A subcommittee consisting of four individuals from HDRN’s AHD-WG was formed (NCH, SB, YZ, and SP). Individuals were selected via volunteer. Members of the subcommittee had expertise in administrative health data, electronic medical record data, chronic disease surveillance, multi-jurisdiction health research, and algorithm development and implementation. Subcommittee members had experience in software programming, quantitative data analysis, and conducting methods research using secondary health data.
Consensus on minimum element identification and item categorization (see
The committee reviewed published algorithm feasibility studies contained within HDRN’s Algorithms Inventory [
Minimum elements were compared to the items contained in the STROBE and RECORD guidelines (see
| 1 | (a) Indicate the study’s design with a commonly used term in the title or the abstract |
RECORD 1.1: The type of data used should be specified in the title or abstract. When possible, the name of the databases used should be included. |
1.a Report the concepts or variables for which algorithm feasibility is assessed. | |
| Background rationale | 2 | Explain the scientific background and rationale for the investigation being reported |
2.a Report the organization/ |
|
| Objectives | 3 | State specific objectives, including any prespecified hypotheses | ||
| Study Design | 4 | Present key elements of study design early in the paper | ||
| Setting | 5 | Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection |
||
| Participants | 6 | RECORD 6.1: The methods of study population selection (such as codes or algorithms used to identify subjects) should be listed in detail. If this is not possible, an explanation should be provided. |
||
| RECORD 6.2: Any validation studies of the codes or algorithms used to select the population should be referenced. If validation was conducted for this study and not published elsewhere, detailed methods and results should be provided. |
||||
| Variables | 7 | Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable | RECORD 7.1: A complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers should be provided. If these cannot be reported, an explanation should be provided. | 7.1.a Describe: (1) algorithm implementation details |
| Data sources/ measurement | 8 | For each variable of interest, give sources of data and details of methods of assessment (measurement). |
8.a Describe key elements of data sources for each jurisdiction including time period covered, population covered, diagnosis code type and number of digits, and maximum number of diagnosis codes per claim. Describe data comparability across jurisdictions. | |
| Bias | 9 | Describe any efforts to address potential sources of bias | ||
| Study size | 10 | Explain how the study size was arrived at | ||
| Quantitative variables | 11 | Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen, and why | ||
| Statistical methods | 12 | (a) Describe all statistical methods, including those used to control for confounding |
12.a Describe and justify the methods selected to assess feasibility, such as face validity. | |
| Data access and cleaning methods | .. | RECORD 12.1: Authors should describe the extent to which the investigators had access to the database population used to create the study population. |
12.1.a Describe methods to address data sharing/ confidentiality. |
|
| Linkage | .. | RECORD 12.3: State whether the study included person-level, institutional-level, or other data linkage across two or more databases. The methods of linkage and methods of linkage quality evaluation should be provided. | ||
| Participants | 13 | (a) Report the numbers of individuals at each stage of the study ( |
RECORD 13.1: Describe in detail the selection of the persons included in the study ( |
13.1.a Describe selection of persons included in the study, stratified by jurisdiction. |
| Descriptive data | 14 | (a) Give characteristics of study participants ( |
14.a Report descriptive |
|
| Outcome data | 15 | |||
| Main results | 16 | (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (e.g., 95% confidence interval). Make clear which confounders were adjusted for and why they were included |
16.a Describe algorithm applied to the study data. If the algorithm was developed or modified from an existing algorithm, provide a rationale for the decisions made that resulted in final algorithm. |
|
| Other analyses | 17 | Report other analyses done—e.g., analyses of subgroups and interactions, and sensitivity analyses | ||
| Key results | 18 | Summarise key results with reference to study objectives | ||
| Limitations | 19 | Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias | RECORD 19.1: Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported. | |
| Interpretation | 20 | Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence |
||
| Generalisability | 21 | Discuss the generalisability (external validity) of the study results |
||
| Funding | 22 | Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based | ||
| Accessibility of protocol, raw data, and programming code | .. | RECORD 22.1: Authors should provide information on how to access any supplemental information such as the study protocol, raw data, or programming code. | ||
*Reference: Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, Sørensen HT, von Elm E, Langan SM; RECORD Working Committee. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015 Oct 6;12(10):e1001885.
STROBE: STrengthening the Reporting of OBservational studies in Epidemiology.
RECORD: REporting of studies Conducted using Observational Routinely collected health Data.
Sufficient items, where the existing STROBE and RECORD items sufficiently covered the details required for transparent reporting of feasibility studies;
Additional context items, where the existing STROBE and RECORD items covered the details required for transparent reporting; however, additional context was required to apply the item to feasibility studies;
New items, where the minimum element was not included in the STROBE or RECORD items. Explanations were provided along with references to justify inclusion of new items.
Subcommittee members compiled the final list of items (sufficient, additional context, and new) and explanations for new items into a report and presentation. The presentation was given to members of HDRN Canada’s AHD-WG and members were invited to comment or provide feedback during a WG meeting. The written report was also distributed to the AHD-WG and members were given three weeks to provide any additional comments. AHD-WG members were asked to review the content, as well as assess for readability and interpretation. Comments were incorporated into the report by NCH with other subcommittee members providing input. The final report was approved by all members of HDRN Canada’s AHD-WG.
Review of published feasibility studies alongside the current STROBE and RECORD guidelines found that the STROBE and RECORD guidelines were sufficient in many cases, particularly in the Title and Abstract, Introduction, and Discussion sections. Six additional context items were identified in the Introduction, Methods, and Discussion sections. Limitations of the STROBE and RECORD guidelines primarily pertained to methods for standardizing heterogenous data across different jurisdictions, algorithm details, and reporting overall and stratified algorithm results. Eleven new items for reporting multi-jurisdictional feasibility studies were identified to address these limitations.
The new items for reporting algorithm feasibility studies recommended by HDRN Canada’s AHD-WG are included in
Item 1.a: Report the concepts or variables for which algorithm feasibility is assessed.
Identification of the concept or variable (e.g., dementia, healthcare service costs, diabetic related hospital visits) relevant to the algorithm(s) in the title or abstract will help other researchers to find the algorithm in published literature. If the feasibility of algorithms for multiple concepts or variables are being assessed, authors should clearly state each individual concept or variable.
STROBE Item 2: Explain the scientific background and rationale for the investigation being reported.
The STROBE guidelines advise the authors to “explain the scientific background and rationale for the investigation being reported”. Authors should highlight the rationale for investigating the algorithm(s), the intended usage, if the algorithm(s) has been applied or validated in any setting previously, and relevant results and references for previous algorithm application or validation.
Item 2.a: Report the organization/system, project, or program behind the study, if applicable.
Surveillance systems and research projects or programs that conduct distributed or federated analyses often conduct algorithm feasibility studies first to ensure the variables of interest can be obtained across different jurisdictions or health databases [
STROBE Item 5: Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection.
The STROBE guidelines advise authors to “Describe the setting, locations, and relevant dates”. A feasibility study spans multiple jurisdictions, and therefore authors should highlight what jurisdictions are included in the study and why the jurisdictions were selected (i.e., data availability, funding, disease or condition of particular interest to those jurisdictions, etc.).
RECORD Item 6.1: The methods of study population selection (such as codes or algorithms used to identify subjects) should be listed in detail. If this is not possible, an explanation should be provided.
The RECORD guidelines advise authors to report “the methods of study population selection (such as codes or algorithms used to identify subjects)”. Healthcare coverage is often a consideration when using electronic health data, therefore authors should highlight any healthcare coverage requirements for inclusion in the study population.
Item 7.1.a: Describe: (1) algorithm implementation details across jurisdictions, including type of data source, fields used from each data source, timeframe/observation window, lookback window details; (2) any specifications such as diagnosis/medication/procedure code(s), number of contacts required, or other definition details; and (3) any other specifications, such as fee/payment codes for a service (delivered or received) or location codes specifying a service location. Where the algorithm includes multiple healthcare contacts or other longitudinal considerations, state how the onset date/classification date is defined (i.e., date of first contact or contact where criteria are fulfilled).
The RECORD guidelines advise “a complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers. . .”. As feasibility studies focus on applying the algorithm and assessing feasibility across multiple jurisdictions, it is recommended that all algorithm elements be provided for each jurisdiction. Elements of an algorithm to be reported (type of data source, observation window, diagnostic/medication/procedure code(s), and number of contacts for each data source) are based on previous publications on algorithm development [
Item 8.a: Describe key elements of data sources for each jurisdiction including time period covered, population covered, diagnosis code type and number of digits, and maximum number of diagnosis codes per claim. Describe data comparability across jurisdictions.
The quality of routinely collected data for research may be impacted by differences in healthcare systems across jurisdictions [
Variations in the data across jurisdictions can impact algorithms’ accuracy and implementation [
Item 12.a: Describe and justify the methods selected to assess feasibility, such as face validity.
Feasibility studies aim to apply algorithms across multiple jurisdictions and assess their face validity. There is no single method or approach to assess face validity; authors should therefore provide justification for why the selected face validity method or approach is appropriate. Examples of methods or approaches to assess face validity can be found in the
Item 12.1.a: Describe methods to address data sharing/ confidentiality.
Authors should report methods for addressing data sharing/ confidentiality across the study jurisdictions. For example, the Canadian Chronic Disease Surveillance System (CCDSS) uses a distributed data analysis system, where algorithm implementation methods (i.e., statistical code) are developed centrally and distributed among participating jurisdictions to be applied locally. Aggregated results are then shared back to the CCDSS for reporting [
Item 12.1.b: Describe methods for aligning data sources/ measurements/variables across jurisdictions (e.g., crosswalk algorithms).
Applying common algorithms across multiple jurisdictions may require methods to standardize or align the data sources. Authors should describe any methods used, such as crosswalk algorithms (i.e., method to convert data from one coding type/version to another coding type/version) [
Item 13.1.a: Describe selection of persons included in the study, stratified by jurisdiction.
Selection of persons may be different across jurisdictions. Authors should include the information required by the STROBE item 13 and RECORD item 13.1 stratified by jurisdiction.
Item 14.a: Report descriptive data for study population, stratified by jurisdiction.
Algorithm feasibility may be influenced by population characteristics across jurisdictions. Therefore, authors should include descriptive information required by STROBE stratified by jurisdiction.
Item 16.a: Describe algorithm applied. If the algorithm was developed or modified from an existing algorithm, provide rationale for the decisions made that resulted in the final algorithm.
Authors should provide a full description of the algorithm that was applied across jurisdictions in the feasibility study. This information should encompass information about the type of data source, timeframe/observation window, lookback window details, diagnostic/medication/procedure/fee/location code(s), and number of contacts required for each data source. This may be similar to the methods section that reported on algorithm implementation. As the main objective for feasibility studies is the implementation of common algorithms across jurisdictions, descriptions of the applied algorithms should be reported as a main result. Where multiple algorithms were considered, only the ones that were applied to the data should be reported as a result. In addition, authors should provide rationale or justification for the choices made during the algorithm development process, if applicable.
Item 16.b: Report results from any face validity assessment of the algorithm by jurisdiction.
Authors should report results stratified by jurisdiction, in addition to overall results (if applicable). If multiple algorithms were assessed, results should be reported separately for each algorithm. Reporting stratified results allows readers to assess the application of the reported algorithms across different populations/databases and its variability and generalizability.
RECORD 19.1: Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported.
RECORD guidelines advise authors to “discuss the implications of using data that were not created or collected to answer the specific research question(s)”. Authors should highlight the implication of using data from multiple jurisdictions and how differences in data availability may impact algorithm application and observed feasibility.
STROBE Item 20: Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence.
STROBE guidelines advise authors to “give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence”. Authors should highlight known jurisdictional differences (e.g., difference in healthcare population coverage, different healthcare delivery models) that could explain or provide insight on observed discrepancies in algorithm performance across jurisdictions. Authors should also state any recommendations on future algorithm development, validation, or implementation, if applicable.
STROBE Item 21: Discuss the generalisability (external validity) of the study results.
STROBE guidelines advise authors to “Discuss the generalisability (external validity) of the study results”. When discussing generalisability, authors should highlight potential challenges associated with algorithm feasibility over time and/or across geography or context, and how this may affect the long-term feasibility of the algorithm. Routinely collected health data are susceptible to data drift [
Blais C, Dai S, Waters C, et al. Assessing the burden of hospitalized and community-care heart failure in Canada. Can J Cardiol 2014; 30: 352–358.
Robitaille C, Bancej C, Dai S, et al. Surveillance of ischemic heart disease should include physician billing claims: population-based evidence from administrative health data across seven Canadian provinces. BMC Cardiovasc Disord 2013; 13: 88.
Marrie RA, Fisk JD, Tremlett H, et al. Differences in the burden of psychiatric comorbidity in MS vs the general population. Neurology 2015; 85: 1972–1979.
Guy P, Sheehan KJ, Morin SN, et al. Feasibility of using administrative data for identifying medical reasons to delay hip fracture surgery: a Canadian database study. BMJ Open 2017; 7: e017869.
Bernstein CN, Kuenzig ME, Coward S, et al. Increased Incidence of inflammatory bowel disease after Hirschsprung disease: a population-based cohort study. J Pediatr 2021; 233: 98-104.e2.
Blais C, Dai S, Waters C, et al. Assessing the burden of hospitalized and community-care heart failure in Canada.
Shiff NJ, Lix LM, Oen K, et al. Chronic inflammatory arthritis prevalence estimates for children and adolescents in three Canadian provinces.
Robitaille C, Bancej C, Dai S, et al. Surveillance of ischemic heart disease should include physician billing claims: population-based evidence from administrative health data across seven Canadian provinces.
Marrie RA, Fisk JD, Tremlett H, et al. Differences in the burden of psychiatric comorbidity in MS vs the general population.
Guy P, Sheehan KJ, Morin SN, et al. Feasibility of using administrative data for identifying medical reasons to delay hip fracture surgery: a Canadian database study.
Godwin M, Williamson T, Khan S, et al. Prevalence and management of hypertension in primary care practices with electronic medical records: a report from the Canadian Primary Care Sentinel Surveillance Network.
Marrie RA, Fisk JD, Tremlett H, et al. Differences in the burden of psychiatric comorbidity in MS vs the general population. Neurology 2015; 85: 1972–1979.
Marrie RA, Garland A, Schaffer SA, et al. Traditional risk factors may not explain increased incidence of myocardial infarction in MS.
Bernstein CN, Kuenzig ME, Coward S, et al. Increased incidence of inflammatory bowel disease after Hirschsprung disease: a population-based cohort study.
Godwin M, Williamson T, Khan S, et al. Prevalence and management of hypertension in primary care practices with electronic medical records: a report from the Canadian Primary Care Sentinel Surveillance Network.
Guy P, Sheehan KJ, Morin SN, et al. Feasibility of using administrative data for identifying medical reasons to delay hip fracture surgery: a Canadian database study.
Marrie RA, Garland A, Schaffer SA, et al. Traditional risk factors may not explain increased incidence of myocardial infarction in MS.
Bernstein CN, Kuenzig ME, Coward S, et al. Increased Incidence of inflammatory bowel disease after Hirschsprung disease: a population-based cohort study.
Godwin M, Williamson T, Khan S, et al. Prevalence and management of hypertension in primary care practices with electronic medical records: a report from the Canadian Primary Care Sentinel Surveillance Network.
Algorithm feasibility studies are intended to assess the application of one or more algorithms for routinely collected health data across multiple jurisdictions. Here, we provide recommendations from HDRN Canada’s AHD-WG on the minimum elements for reporting algorithm feasibility studies. These minimum elements address many of the challenges associated with using data from multiple jurisdictions in observational studies and are intended to aid researchers in describing jurisdictional data variations and efforts to standardize or harmonize data across jurisdictions. Recommendations are meant to augment RECORD and STROBE reporting guidelines and represent the minimum standard for producing reproducible and transparent information about algorithms. By adopting these recommended minimum elements for reporting, future feasibility studies will have improved transparency and better inform algorithm development and reuse. Moreover, providing clearer standards for reporting feasibility studies encourages both researchers and journals to publish algorithm feasibility work. Additional information that goes beyond the minimum requirements, such as open access to code production for algorithm implementation, should also be considered when working towards the requirements of open data and open science.
Strengths of this work are that it was undertaken by Canadian experts in algorithm development and implementation for routinely collected data. As well, this work was led by HDRN Canada, which has taken a leadership role in data harmonization and algorithm standardization across Canadian provinces and territories [
There are some limitations to this work. Articles screened to develop the recommended minimum elements for reporting feasibility studies were limited to health domain articles within HDRN Canada’s Algorithms inventory. The inventory is based on previously published studies, which may inherently reflect specific regions or populations that have been more extensively studied (e.g., provinces like British Columbia and Ontario are more frequently represented). Additionally, studies included in the review were in English only, which could introduce language bias and may have a disproportionate focus on high-income regions. This limits the generalizability of the findings to non-health domain studies, lower-income regions, and non-English language studies. In addition, content expertise of the subcommittee may not have been comprehensive, as there were only four members. To mitigate this, the minimum elements, explanations, and references were reviewed by all members of HDRN Canada’s AHD-WG, which at the time of this work was comprised of more than 14 organizations.
In summary, transparent reporting and publication of feasibility studies is critical for ensuring the scientific rigor of algorithm development and facilitating algorithm reuse. By recommending minimum elements for transparent reporting, HDRN Canada provides a framework for transparent reporting and publication of feasibility studies. HDRN Canada is a nation-wide effort to enable multi-jurisdictional research through harmonized electronic datasets and national data linkages [
The authors would like to sincerely thank the HDRN AHD-WG for the insights and input provided on the recommended minimum elements as well as their explanations.
This work did not require ethical approval as it did not involve primary data collection, secondary data analysis, or non-public record review.
Data for this work was obtained through review of published articles contained within Health Data Research Network Canada’s Algorithms Inventory, at