Supporting policy and practice in Ontario through ICES’ Applied Health Research Question (AHRQ) Program

Abstract

ICES upholds a strong reputation for generating high-quality evidence to inform policy and practice through its collaborations with a broad range of health system stakeholders including government policymakers and healthcare providers including clinicians. Supported by the Ontario Ministry of Health and Ministry of Long-Term Care, the ICES Applied Health Research Question (AHRQ) Program leverages the data holdings and, scientific and clinical expertise to generate evidence tailored to the information needs of requestors. This paper outlines the approach, process, strengths, challenges and the resulting influence and impact to the healthcare landscape in Ontario.

Introduction

Canada has a publicly funded healthcare system wherein each province and territory (P/T) is responsible for providing residents with universal access to an agreed set of tax financed physician services and hospital care that are funded by transfers from the federal government [1]. Since the P/Ts have jurisdiction over the provision and management of health insurance, the data they collect to manage these programs reside within and are controlled by P/T health ministries, but are commonly shared with other organizations such as ICES for the purposes of health system planning and evaluation.

ICES (formerly the Institute for Clinical Evaluative Sciences) is an independent, not-for-profit research institute that uses research and analytics to improve the health and healthcare of over 14 million individuals in Ontario, Canada. Established in 1992, ICES’ work is founded on an array of linked, coded health-related data within a secure environment including service encounters and registration, vital statistics, provider characteristics, registries and population-based surveys. The data include the vast majority of the publicly funded healthcare records of those covered under the Ontario Health Insurance Plan (OHIP). These data reside at ICES as a result of strong partnerships and its status as a Prescribed Entity under the province’s Personal Health Information Protection Act (PHIPA) which is subject to Information and Privacy Commissioner of Ontario (IPC) oversight and renewal tri-annually. ICES is one of four organizations in Ontario to hold the legal designation of Prescribed Entity, which permits health information custodians to disclose to personal health information to ICES without individual patient consent or research ethics board approval for the purposes of health planning, management and evaluation of the healthcare system [2].

ICES upholds its reputation for generating high-quality evidence to inform policy and practice through its collaborations with a broad range of health system stakeholders, including government policymakers, and healthcare providers including clinicians. Supported by annual funding from the Ontario Ministry of Health, the ICES Applied Health Research Question (AHRQ) Program leverages the data holdings and, scientific and clinical expertise to generate evidence tailored to the information needs of Knowledge Users [3]. Eligible AHRQ ‘Knowledge Users’ include government ministries, public health agencies, hospitals, institutions, community groups, and Indigenous communities. ICES data and analytic services are provided free of charge to Knowledge Users through annual grants from the Ontario Ministry of Health. In addition, as part of Ontario’s response to the COVID-19 pandemic, the Ontario Ministry of Health launched the Ontario Health Data Platform (OHDP), a federated data platform with ICES as a key partner, leveraging ICES infrastructure. In supporting the OHDP, ICES expanded its scope of the ICES AHRQ Program to enable rapid support for COVID-related requests from health system stakeholders.

Approach

There are three types of projects that ICES undertakes in response to an AHRQ request: rapid response, research report or technical brief, and research project.

Rapid response

A rapid response request is the quickest and simplest form of AHRQ. This type of AHRQ typically provides Knowledge Users with preliminary information, expert opinion, or material that is readily available, within five business days of a request. These projects are given organization-wide priority at ICES and a project team is quickly mobilized to compile the required data. Only a handful of these requests are received at ICES each year. From an analytic perspective, these projects typically involve linkage among a limited number of datasets to generate a simple table. An example of a rapid response request might be the annual volume of non-urgent emergency department visits at the Knowledge User’s hospital over the last decade to assist with an upcoming board meeting to discuss resource allocation, or the provincial Minister of Health looking for information for a news briefing regarding annual numbers of hospital admissions for a specific condition. Rapid response requests must provide a strong rationale for the need to provide the research evidence expeditiously to address a decision or issue of organizational priority.

Research report or technical brief

Research reports are the most requested type of AHRQ. These benefit heavily from pre-existing ICES analyses. Project duration typically ranges from two to four months. Where possible, ICES will leverage previously validated analytic code and will adjust accordingly to meet the needs of the request. Deliverables may include a set of tables, maps or a brief report and presentation. Analytically, these projects vary greatly in methods and data sources. An example of a research or technical brief would be an examination of healthcare utilization across Ontario, by various geographic areas and demographic stratifications (i.e., age group, neighbourhood income) and mapped to illustrate regional variation. This request was made by a group of healthcare providers interested in identifying healthcare equity gaps across populations and geographies [4].

Research project

Research projects are conducted when new evidence must be generated to answer the question. These projects may involve the importation of external datasets for the purpose of linkage at the individual level, which involve data sharing agreements, and can take upwards of a year to complete. Research projects not involving data importation often require linkage across multiple data holdings to identify a study cohort based upon demographic characteristics, health conditions, and specific health service encounters or interventions. An example of a research project is an comparison of subsequent healthcare utilization between patients enrolled in a new cardiovascular clinic program compared to patients with similar healthcare and demographic profiles who are not enrolled in the program [5]. Similar to deliverables for research reports or technical briefs, deliverables for research projects can include a combination of tables, maps, presentations and full reports.

Application and approval processes

Potentially eligible AHRQ Knowledge Users submit requests to ICES identifying the research question and how the research evidence will help to inform decision-making relevant to the provision of healthcare services in Ontario. The ICES AHRQ Adjudication Committee meets monthly to review requests, with rapid response requests reviewed upon receipt. The Committee consists of the ICES AHRQ Project Manager, two ICES Research Program Leaders, two ICES Staff Scientists, and the ICES Manager, Public Engagement and Knowledge Translation. The request is adjudicated based on several criteria including whether the requestor represents an eligible Ontario-based organization; the purpose of the request; whether the question can be answered with ICES data (versus by another Ontario AHRQ Research Provider [3]); the extent to which the evidence generated is likely to influence service provision, policy or program development, as proposed; and whether the evidence is likely to be generalizable and useful to Ontario decision-makers more broadly. The request must demonstrate plans to disseminate the findings and how the information will be used by the organization to improve healthcare services, delivery or outcomes. Requests must not be designed specifically to promote or advocate for a constituency or position, for marketing purposes or for the purpose of preparing applications for academic research funding. The AHRQ Program is not intended to support work that is fundable through usual mechanisms for academic research such as peer-reviewed research grants. Over the past eight years, ICES has received a steady number of requests without direct solicitation and consistently exceeds the Ontario Ministry of Health’s annual target of 25 new requests (Figure 1).

Figure 1: Types of AHRQ requests submitted by year. Rapid responses provide Knowledge Users with information that is readily available often within five business days of a request; Research reports or Technical briefs leverage pre-existing ICES analyses and range from two to four months to complete; and Research projects often involve data importation or the creation of new information and can take up to a year to complete.

Once a request is approved by the AHRQ Adjudication Committee, the Knowledge User is notified and invited to have a discussion with ICES staff to enable the planning of the analytics proposed (Figure 2). Outside the rapid response category, requests are undertaken mostly on a first-come, first-served basis with some flexibility to allow Knowledge Users to request a timeline that best meet their decision-making needs, and we do our very best to accommodate them. The request is assigned to an ICES Scientist with expertise in the subject area to lead the project. The Knowledge User details their approved question and ICES staff begin to draft an analytic plan. This plan identifies the datasets that will need to be accessed and linked, specific permissions from data partners, and whether any further consultations with community groups or other organizations are needed for the project to commence. In addition, ICES updates its public website to include a short description of the request including the name of the requesting organization, assigned ICES scientific lead, purpose of request and project status [6]. This information is updated on a quarterly basis to keep an on-going public log of all AHRQ requests and deliverables.

Figure 2: ICES AHRQ program process flow. This flow diagram outlines the general steps that take place once a Knowledge User submits a request to ICES to when the project is considered complete.

Prior to data access, each AHRQ project submits a privacy impact assessment which is reviewed and approved by the ICES Privacy and Legal Office, ensuring the study meets the purpose of evaluation, management and planning of the healthcare system and ICES’ policies to uphold privacy and security standards are met (Table 1).

Table 1: The ICES project privacy impact assessment process. Every project undertaken at ICES must have an approved ICES Project Privacy Impact Assessment. The table above provides a high-level understanding of the components involved in reviewing and approving a project, to ensure that it meets the strict privacy requirements.

ICES project privacy impact assessment

• Specify the purposes for which a project proposes to collect, use and disclose information (personal and non-personal)

• Document the individuals on a project who require access to information

• Document all information to be collected, used and disclosed for the project

• Ensure that only personal information that is necessary and relevant for the project’s specified purpose and objectives will be used for the project

• Determine the legal and best practices framework in which collection, use and disclosure of information can occur

• Ensure that requirements for collection, use and disclosure of information have been met

• Review any agreements and research ethics board applications and approvals that could impact collection, use or disclosure of information for the project

• Review the privacy implications of any provider-level reporting and ensure appropriate permissions are in place

• Identify risks associated with conducting a project and determine a mitigation plan

• Document approvals of all required reviewers including ICES Program Leader, ICES Privacy & Legal Office (and Chief Science Officer, as required)

AHRQ requests are assigned to a project team which include an ICES scientist(s), project manager, epidemiologist and research analyst, specifically composed to respond best to the research question asked. Additional leveraging of the ICES community of subject matter experts and specialized expertise (e.g., statistical methodologist or medical geographer) is called upon on an as needed basis. The project team is responsible for finalizing the analytic plan which is discussed with and approved by the Knowledge User and for carrying out the analyses. There is regular communication with the Knowledge User during the analysis phase to ensure that emerging findings support their request. The Knowledge User plays a critical role in helping to interpret the findings. Interim deliverables are provided regularly and every effort is made to provide to the Knowledge User the final deliverable prior to their desired date.

Reports, tables or presentations are customized to support the Knowledge User’s plans for informing their policy, planning and decision-making purposes. Additional support is provided if methods or results require clarification. Once the Knowledge User has confirmed that the deliverables have met their needs, ICES updates the project information on its website to indicate completion [6] and members of the public can request a copy of the final deliverable. It is the Knowledge User and their organization’s responsibility to effectively apply and disseminate the findings according to their plans, this may or may not include a publication. However, ICES reserves the right to publish the findings of AHRQ projects. Knowledge Users and ICES project teams frequently publish their findings as a final objective prior to project closure. While there are no restrictions on how Knowledge Users choose to share their findings, the expectation is that it will be aligned with their intentions expressed within the accepted request.

Examples of high-impact AHRQ projects

Since 2014/15, the AHRQ Program has accepted over 300 AHRQ requests, ranging from 44 to 61 each year (Figure 1). Project summaries are available on the ICES website [6] with more details available from the AHRQ Program on request. Although numerous AHRQ projects have had important impact upon policy, program design and program funding, several key areas of emphasis stand out as being particularly important.

Supporting the transformation of healthcare services and delivery in Ontario

In 2019, the Ontario government launched the formation of Ontario Health Teams (OHTs); these multi-disciplinary teams of health care providers (including hospitals, doctors and home and community care providers) work as one coordinated team to address the specific needs and provide coordinated care to patients within their community [7]. The first OHT-related ICES AHRQ project was integral to the initial formation and leveraged methodology from Physician Networks developed at ICES [8] to define the attributed population for all OHTs [7]. To encourage targeted care in the initial phase of team development, emerging OHTs were recommended to identify a subset of their attributed population to concentrate care transformation and improvement efforts before submitting a full application to the Ontario Ministry of Health. The ICES AHRQ Program helped to identify these subpopulations to support applications. For example, the North Rideau Health Alliance OHT (now referred to as the Four Rivers OHT) leveraged the ICES AHRQ Program to characterize their patient population by age groups, local geography and chronic health conditions. Specifically, the request was interested in identifying the proportion either receiving care for or diagnosed with a mental illness, a subpopulation the team has decided to focus on improving services and outcomes for in their first year of development [9]. Along with responding to specific requests posed by individual OHTs, the AHRQ Program also enables other teams like the Health System Performance Network to provide support to the OHTs by making accessible, with appropriate approvals, the attributed population files and associated updates that flows to ICES from the Ministry of Health [10].

Response to COVID-19

Starting in early March 2020, ICES began receiving many requests related to the COVID-19 pandemic. Early requests were related to informing surge capacity planning in hospitals including identifying “elective” surgeries which result in intensive care unit (ICU) hospitalization and ventilator use. By mid-May 2020, there was interest in small area-level data on COVID-19 testing and test recipient characteristics. To further enhance the data that ICES could provide to Knowledge Users, ICES engaged with data partners to negotiate and sign new data sharing agreements for COVID-19 laboratory data as well as others. Knowledge Users included the Ontario Ministry of Health, the Ontario Ministry of Long-Term Care, Public Health Ontario, Indigenous organizations and local public health units. Adjudication and project initiation processes for projects became an organization-wide priority and were expedited. Projects were given the option of regular reporting (daily, weekly, biweekly or monthly) as the need to access information was required urgently to inform regional and system-wide planning and decision-making. To present day, ICES continues to provide regular updates to a variety of Knowledge Users as part of our contribution to the pandemic response and have also been publicly reporting testing and vaccination indicators, as well as providing downloadable data files on ICES’ COVID-19 Dashboard [11] in response to public interest [12–14]. In addition, direct access to encrypted individual level data was provided to external researchers who were approved to be “non-affiliated ICES agents”. Novel secure data access mechanisms and time-limited authorizations were developed to enable trusted subject matter experts with expertise in infectious disease modelling to support government-appointed advisory tables. These external researchers continue to model scenarios based on a number of evolving factors, helping to inform government and policymakers of the possible outcomes as a result of lockdown or easing restrictions.

Discussion

Strengths

ICES is notable in its holdings of a vast array of population-based data coupled with strong scientific expertise in how the data can be used for research, as well as its access to a cadre of ICES scientists with clinical and health policy expertise. As such, the ICES AHRQ Program can address a broad array of health system requests and facilitate external data linkage, something only select Ontario organizations are legally permitted to do. Unique to ICES, the organization has a linked repository of validated algorithms and coding macros, which speaks to the long history of conducting health services research and broader sharing of science within the organization. The ICES AHRQ Program fosters ongoing, iterative discussion with Knowledge Users to ensure that analysis plans are contextually appropriate and that deliverables meet knowledge translation needs. This engagement is especially important for projects involving Indigenous communities and organizations.

Within the ICES AHRQ Program, the Indigenous Portfolio works directly with Indigenous partner organizations and communities and their respective leaders, to provide answers to questions directly affecting their populations, and tailor the presentation of results to suit their specific needs. Within this framework, there are key partnerships and governance agreements in place to ensure Indigenous data is only accessed where appropriate and aligns its processes with the OCAP^® (Ownership, Control, Access, and Possession) [15] principles of First Nations data governance. The Indigenous Portfolio also ensures that information contained in projects without an Indigenous focus, do not identify specific First Nation communities without their expressed approval and participation. As such ICES plays an active role in ensuring that organizational and/or community leaders are heavily involved throughout the process and the resulting data are tailored to suit the specific needs of the Indigenous partners.

Finally, while the AHRQ Program routinely operates at full capacity, ICES takes every opportunity for outreach to prospective Knowledge Users including presentations at conferences and other relevant forums and the distribution of promotional materials, including a public-friendly flyer to explain the program and eligibility. In recent years, ICES has actively sought feedback from Knowledge Users to gauge satisfaction and experience in working with the AHRQ Program. One specific improvement that resulted from Knowledge User feedback has been improved stakeholder engagement on the part of the AHRQ Project Manager and Staff Scientists throughout the course of projects. Both roles have been critical to ensuring steady progress and continuity. As the needs and priorities of Knowledge Users evolve, it is important for ICES to keep a pulse on how the requesters themselves are using the AHRQ Program, how it is helpful and how improvements can be made to current processes.

Challenges

The ICES AHRQ Program experiences challenges which are frequently linked to data access, linkage and timeliness of the data received from data partners. Some requests are not feasible due to the limitations of the ICES data holdings or the timeliness of the provincial administrative data feeds. ICES receives updates to its core data holdings at frequencies ranging from monthly to annually. For example, requests requiring real-time hospitalization or mortality data are challenging due to data lags. Where possible, efforts are made to determine whether data sharing agreements can be negotiated or amended to accommodate data access for a request. COVID-19 AHRQs had a direct impact upon the frequency of our core data feeds as data sharing agreements were amended and new agreements struck to make more rapid work possible. AHRQ projects requiring importation of externally collected data for the purposes of linkage can also experience delays with executing the data sharing agreements and data transfer, as well as subsequent challenges that may occur with probabilistic linkage to ICES’ data. As well, since AHRQ requests must meet criteria for policy relevance and impact on healthcare delivery or service, requests may not be eligible for AHRQ funding and other ICES service streams are proposed to the requestor for a fee. In addition, requests with large scope may need to be readjusted to ensure equitable distribution of funding and resources across projects each year. Similarly, Knowledge Users who submit multiple requests may have subsequent or lower priority projects placed on hold or deferred.

Future plans

It is a priority for ICES to ensure that the ICES AHRQ Program is relevant and continues to meet the needs of the community and health system leaders. The COVID-19 pandemic reinforced the importance of being nimble and responsive so as to best meet the evolving information needs of Knowledge Users.

Recently, ICES sought feedback from the ICES Public Advisory Council to understand how to make the ICES AHRQ Program more responsive to community needs. ICES Public Advisory Council members have identified the importance of making the ICES AHRQ Program more accessible to the general public. They selected specific projects to further highlight and to make available the respective deliverables in an easily digestible format. As such, work is currently underway to ensure that members of the public have a process through which they can request project deliverables and are aware of key findings produced by the ICES AHRQ Program. ICES also plans to work with its Public Advisory Council to develop a research question. By using the AHRQ Program to answer a question generated directly by public members, ICES aims to demonstrate transparency around the use of public funding and better understand the public’s interests and priorities for health data research.

Next, the Indigenous Portfolio broadens its reach and engagement with more communities across the province, being responsive to these partners includes building a process tailored to the special needs for engagement with Indigenous communities. Work is currently underway to develop a separate application and approval process through which Indigenous partners can apply and have project results shared in a manner that maximizes benefit to communities.

And finally, while ICES has a long standing reputation of collecting health administrative data for the purposes of evaluation, management and planning of the healthcare system, the data has historically been health-focused. As the reach of the AHRQ Program grows to support different branches of government ministries, there is increasing interest in non-health data including social assistance, education, race, ethnicity and other sociodemographic information. As a result, new partnerships are forged to respond to new questions and allow for the data to flow to ICES for the purposes of linkage to provide a fulsome understanding of how these factors affect healthcare utilization. Specifically, ICES is considering other ways to leverage the AHRQ Program to support initiatives related to equity and research, specifically in response to ongoing efforts to ensure the appropriate use of race and ethnicity data in health research. ICES is investigating how the AHRQ Program can be used to support equity-seeking community organizations focused on addressing systemic racism to rally for and inform change for their communities.

Conclusion

It has been a priority for ICES to provide research evidence to support health system stakeholders for policy, planning and program development through the ICES AHRQ Program. Since inception, funding from the Ontario Ministry of Health has enabled ICES to provide data and analytic services for this specific purpose.

The ICES AHRQ Program has shown tremendous value in supporting positive healthcare policy and practice changes in Ontario. AHRQs are meant to serve as a vehicle through which eligible Knowledge Users can get answers to important questions for the purposes of evaluating, planning or managing Ontario’s healthcare system. The active interaction that occurs when Knowledge Users with important contextual knowledge, partner with ICES’ data and science expertise, has a profound impact on the resulting work. ICES engages directly with Knowledge Users to fully understand the question being asked and develop a tailored analytic plan to yield useful results customized for their specific use. The examples highlighted illustrate the reach and power of the program to directly inform decision-making at local, regional and provincial levels.

Now, more than ever before, it is critical to provide timely and accurate information to Knowledge Users who will shape and transform how healthcare services and delivery are tailored to the Ontario landscape – and ICES is well positioned to work directly with Knowledge Users to influence both policy and practice.

Acknowledgements

ICES is an independent, not-for-profit organization, funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by the MOH or MLTC is intended or should be inferred. ICES acknowledges the Ontario Health Data Platform (OHDP), a Province of Ontario initiative to support Ontario’s ongoing response to COVID-19 and its related impacts. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by the OHDP, its partners, or the Province of Ontario is intended or should be inferred.

Ethics statement

ICES is a prescribed entity under section 45 of Ontario’s Personal Health Information Protection Act (PHIPA). Section 45 authorizes ICES to collect personal health information, without consent, for the purpose of analysis or compiling statistical information with respect to the management, evaluation, monitoring or planning for all or part of the health system. Applied Health Research Question (AHRQ) projects are typically conducted under section 45 of PHIPA and do not require review by a research ethics board.

Declaration of conflicts of interest (COI)

All authors are employees of ICES, which is funded by the Ontario Ministries of Health and Long-Term Care.

References

1. Government of Canada. Canada’s Health Care System. Reports and Publications - Health Care System. [Online] August 5, 2021. https://www.canada.ca/en/health-canada/services/health-care-system/reports-publications/health-care-system/canada.html.

2. Information and Privacy Commissioner of Ontario. PHIPA - Information and Privacy Commissioner of Ontario. Frequently Asked Questions - Personal Health Information Protection Act. [Online] September 2015. https://www.ipc.on.ca/wp-content/uploads/2015/11/phipa-faq.pdf.

3. Ontario Ministry of Health. Applied Health Research Question Program. Research at the Ministry of Health. [Online] May 17, 2021. https://www.health.gov.on.ca/en/pro/ministry/research/applied_heq.aspx.

4. Ontario Community Health Profiles Partnership. Ontario Community Health Profiles Partnership. Ontario Community Health Profiles Partnership. [Online] May 17, 2021. http://www.ontariohealthprofiles.ca/.

5. Importance of completing hybrid cardiac rehabilitation for long-term outcomes: a real-world evaluation. Suskin NG, Shariff SZ, Garg AX, Reid J, Unsworth K, Prior PL, Alter D. 2019 Feb 28; 8(3), J Clin Med, p. 290.

6. ICES. AHRQ Projects. AHRQ. [Online] May 7, 2021. www.ices.on.ca/DAS/AHRQ/AHRQ-Projects.

7. Ontario Ministry of Health. Become an Ontario Health Team. Ontario Ministry of Health. [Online] May 21, 2021. https://health.gov.on.ca/en/pro/programs/connectedcare/oht/.

8. Multispecialty physician networks in Ontario. Stukel TA, Glazier RH, Schultz SE, Guan J, Zagorski BM, Gozdyra P, Henry DA. 2, May 14, 2013, Open Medicine, Vol. 72, pp. 40–55.

9. Kemptville District Hospital. GREEN LIGHT FOR ONTARIO HEALTH TEAM APPLICATION. Kemptville District Hospital. [Online] May 14, 2021. https://www.kdh.on.ca/news-releases/green-light-for-ontario-health-team-application/.

10. Health System Performance Network. OHT Focus Populations: Ideas and Measures. Health System Performance Network. [Online] April 27, 2021. https://hspn.ca/wp-content/uploads/2021/04/HSPN_webinar_targetpops_april27_post.pdf.

11. ICES. ICES COVID-19 Dashboard. ICES. [Online] May 31, 2021. https://www.ices.on.ca/DAS/AHRQ/COVID-19-Dashboard.

12. Toronto: ICES; 2021.
13. Toronto: ICES; 2021.

14. The role of a ‘resilient information infrastructure’ in COVID-19 vaccine update in Ontario. Duchen R, Iskander C, Chung H, Paterson JM, Kwong J, Bronskill SE, Rosella R, Guttmann A. 2021, Healthcare Quarterly.

15. First Nations Information Governance Centre. Understanding OCAP® The First Nations Principles of OCAP®. [Online] May 17, 2021. https://fnigc.ca/ocap-training/.