Closing the loop: From system-based data to evidence-influenced policy and practice
Main Article Content
Abstract
For more than 30 years, the Manitoba Centre for Health Policy has been conducting research and evaluation to provide timely and critical evidence to answer real-world policy questions. Our experienced team of research scientists, analysts and other staff work extensively with policy-makers at the macro, meso and micro levels of government to support evidence-informed policy and program development in an effort to ensure that policy initiatives provide the greatest benefit possible to individuals and society as a whole. Using the widely recognized whole-population Manitoba Population Research Data Repository, which comprises approximately 100 different datasets from multiple sectors, we employ sophisticated and state-of-the-art research methods and data science technologies, and then translate the results into meaningful insights or recommendations for policy-makers.
Our long and productive history of working with policy-makers has taught us much about making our research relevant to policy-makers. In this article, we outline some examples of how research evidence has been used to influence policy in Manitoba, and the key lessons we have learned about what makes relationships between researchers and policy-makers work. In essence, policy-makers have supported the growth of the Repository over the last 30 years, because researchers have "closed the loop" by sharing valuable and policy-relevant research results with them. This ability to inform policies, programs and service delivery with scientific evidence continues to benefit individuals, communities and our society as a whole.
Introduction
The Manitoba Centre for Health Policy: using whole-population administrative data to influence policy-making
Health and social policy-making is a key part of the mandate of those who design, manage and lead public services at the macro, meso and micro levels. In Canada, the federal government (macro level) influences how federal policies are implemented in provincial and territorial systems, for example, by providing fiscal transfers to support health and social services. At the meso level, the provinces and territories are responsible for designing and managing health services and most social programs, but often delegate implementation to local (micro level) system managers and service providers, including Regional Health Authorities or Boards, School Boards, Child Welfare Authorities and other similar groups.
Influencing policy in Canada’s distributed provincial system, in which local and regional policies and circumstances dictate different approaches to common challenges, is complex. The players responsible for developing policies and those that manage and implement them are often not optimally coordinated. Data limitations and resource constraints make it challenging to evaluate policies and programs to ensure they are meeting the needs of the intended recipients. The often ‘siloed’ nature of policy development and service delivery can result in lost opportunities to coordinate services for individuals, families and communities whose needs span multiple departments. There is also political and public pressure to ensure optimal use of available resources. Rigorous research and evaluation to support evidence-informed policy and program development can help to ensure that resources invested in these initiatives provide the greatest benefit possible to individuals and society as a whole.
For more than 30 years, the Manitoba Centre for Health Policy (MCHP) has been conducting research and evaluation to provide timely and critical evidence to answer these types of real-world policy questions using our world-class whole-population Manitoba Population Research Data Repository. The Repository is a powerful data resource, and serves as a platform for sophisticated intersectoral analyses that can impact the design and delivery of new and existing policies and programs [1, 2]. The vast range of data in the Repository is a testament to the strong and varied relationships MCHP has developed with leadership at all three policy levels. Early relationships were predominantly within the healthcare system [3–5], but a growing awareness of the importance of the social determinants of health [6, 7] has led us to develop new relationships with other departments, and has expanded our research horizons to include social services [8, 9], education [9, 10], justice [11] and housing [12, 13]. Our data providers value the evidence we generate from their data to support their work: government departments who share their whole-population administrative data with MCHP use our analyses to support policy development and evaluation, and healthcare managers and providers make their clinical data available to enhance their research and provision of care [1, 2].
MCHP comprises a team of driven, talented and innovative research scientists and staff, and we collaborate with numerous affiliated scientists and scholars in and beyond Manitoba, as well as with First Nations, Métis and Inuit organizations and research partners. Jointly, we have extensive expertise in many research disciplines and work sectors, in managing and analyzing administrative, survey and registry data, in applying state-of-the-art research methods to answer policy questions, and in interpreting the research results into practical policy-relevant approaches, strategies and solutions.
Good public policy supports democracy and enables citizens to realize their full human potential. Our underlying philosophy is to provide high quality evidence for the public good across a broad range of sectors and research areas. We strive to build research capacity and develop strategies to ensure citizens in Manitoba and Canada receive optimal benefit from new and existing health and social policies and programs.
Approach
Our tools: the Manitoba Population Research Data Repository
The central component of the Repository is the Manitoba Population Registry. The registry includes demographic information on all Manitobans registered for healthcare insurance through Manitoba’s universal publicly funded healthcare system (representing >99% of Manitobans). The only residents not included in this registry are individuals who are under federal jurisdiction (e.g., military personnel, members of the Royal Canadian Mounted Police, and individuals incarcerated in federal prisons). Analyses using the Repository are therefore based on virtually the entire population, rather than only those who accessed services or responded to surveys. The Registry also serves as the key for linking data at the individual level across different datasets and over time. Before any data are sent to MCHP, the Ministry of Health in the Manitoba Government removes all identifying information (such as names and addresses) from each record, and then attaches an encrypted 9-digit personal health identification number. Because this numeric identifier is encrypted in the same way for each dataset, it serves as a link across records from multiple datasets and over time while protecting the privacy of Manitobans’ personal health information. This process has been previously described in more detail [1, 14–17].
At founding, the Repository housed 14 databases – currently, it encompasses approximately 100 linkable databases across six domains and describes health and health service use, education, social services, and justice system involvement, and also includes the associated registries and data support files. It is updated regularly: some datasets are updated weekly, others monthly, and most are updated annually. Significant resources are devoted to ensuring that data quality is maintained from data collection to the linkage process [18, 19].
Our capacity for repository-based research: analyst and scientist expertise
The journey from data to information to the knowledge required to influence policy is fraught with challenges. This process, what we call “analysis”, requires adaptability, creativity, critical thinking and a firm commitment to careful problem-solving. We and others have previously described the many pitfalls those not familiar with administrative and registry data and their strengths and weaknesses could encounter [20–22]. But in the context of conducting analyses to inform policy, we consider the close working relationship between our highly skilled and experienced analysts and our research scientists to be immensely important. Administrative data research meant to influence policy requires a policy question that the data can address, an understanding of how to apply cutting-edge analytic methods, statistical modelling and data science technologies, and analysts who can successfully execute the analyses in a complex data environment to address the question [23–25]. The research scientist role is to frame policy-maker queries and priorities as answerable questions, work with the analysts to communicate those questions and develop an appropriate analytic approach, and, once the analyses are complete, translate the results into meaningful insights or recommendations for the policy-makers.
Our knowledge translation approach: turning evidence into policy
There are many factors that can influence and shape the decision-making process. Figure 1 shows some of the perspectives or “lenses” a policy-maker may consider when assessing a policy proposal, all of which contribute to how the policy-maker views the issue and the final decision they make. The evidence MCHP produces can be an important part of this process.
At MCHP, our long and productive history of working with policy-makers has taught us much about making our research relevant to policy-makers [27, 28]. One long-standing, illustrative example of the lessons we have learned and how we continue to apply them is our work with the Need to Know (NTK) Team [29]. The NTK Team is an innovative knowledge translation approach initially funded in 2001 by the Canadian Institutes of Health Research, but its remarkable success has resulted in long-term funding from MCHP’s operations budget. It provides an ongoing platform for two-way communication between scientists and analysts at MCHP and regional health planners in Manitoba. At regular meetings and workshops, the community-based health planners share the questions they would like to have answered, and the researchers and analysts build capacity among them for understanding what the data can address and how. Helping the health planners understand our scientific methods and approaches supports greater acceptance of the results and what they mean for the health regions, while the researchers benefit from the insights of planners. During an NTK Team renewal that coincided with the recent health system restructuring in Manitoba, members of the team emphasized how important their experience with the NTK Team was in integrating research findings into policy development.
Major lessons MCHP and other population data centres have learned about working with policy-makers over the years were summarized in a paper by former MCHP directors. Martens and Roos (2005) describe the interactions between researchers and decision-makers as being like “tectonic plates” [27]. Sometimes the tectonic plates move slowly past each other with little noticeable change to the landscape, but other times there is a great deal of friction, resulting in major tidal waves or volcanic eruptions on the policy scene. Many of those lessons remain relevant today. We present here an updated overview of our approach to working with policy-makers that ensures we build credibility within the real-world realm of policy-making, and thereby improve the likelihood that our evidence will be understood and will lead to policy action.
Building relationships
Establishing relationships is one of MCHP’s major successes, because these relationships open doors for various types of research questions, research opportunities and knowledge translation initiatives. The relationships we develop may be formal contractual relationships or more casual collaborations, but in all cases, their success is dependent on trust and respect amongst all parties. Relationship building takes time and requires good communication amongst participants. Researchers must make major investments in building relationships with policy-makers to overcome the inevitable tensions between what the two parties need from the relationship and what they do to achieve their goals.
Building and maintaining meaningful relationships with Indigenous organizations has been crucial to MCHP research focused on influencing policies that address the longstanding impacts of colonialism and racism ubiquitous in Canadian society. When we engage in research with Indigenous Peoples, we follow the underlying principle of “nothing about us without us”, and we involve Indigenous partners from the very earliest stages of the research through to the end of the project. We recognize their data sovereignty over their own health information, and work closely with them to ensure that the research aligns with OCAP™ (Ownership, Control, Access and Possession) [30] or OCAS (Ownership, Control, Access and Stewardship) [31] principles and that the research results will benefit their health and wellbeing.
Navigating political priorities
Differences in political perspectives create many challenges for research-government partnerships. Over the last two decades, MCHP has operated under the leadership of two different political parties in Manitoba, and has had to navigate frequent shifts in policy-maker priorities in order to continue to have a positive impact on evidence-based policy and planning [27, 32]. For example, some years ago, the Minister of Health responded negatively to a government-contracted MCHP report prior to it being made public. Based on a longstanding, respectful relationship with government officials, the MCHP leadership was able to negotiate a mutually acceptable solution that respected the academic freedom of the research team while addressing the Minister’s concerns. Researchers need to be aware that their words and actions can impact both new and more established relationships with policy-makers. Often the issues that arise can be overcome with goodwill, clear communication and patience. Trust is the glue that holds those relationships together, but can be easily jeopardized.
Honing the message
Two-way communication is key to relationship building, but a strategic approach for what needs to be communicated and how it is communicated, is what makes it work. A researcher’s job is to figure out and communicate the meaning of their results – we often do this by publishing scientific manuscripts or presenting to other researchers at conferences. But when it comes to influencing policy, we need to be able to explain our complex methods and nuanced results in a way that non-scientists understand. MCHP’s experience in briefing cabinet ministers and deputy ministers has taught us much about the value of being able to communicate our research findings in a concise, action-oriented and policy-relevant way [28, 29].
Policy-makers are continually presented with “solutions” to problems that require complex interactions between multiple factors. The messaging from researchers needs to capture their attention convincingly. MCHP has had good success with the use of analogies, stories and even music. For example, our previous director, Dr. Pat Martens, famously compared the Repository to the ubiquitous “Little Black Dress” [33], an adaptable piece of clothing with the potential to be worn appropriately in many different circumstances and accessorized in different ways. In the same way, the Repository has multiple uses and can potentially provide the answers to many different questions when used correctly. And while the metaphor may not resonate with everyone, it certainly is memorable and gets the message across.
Although we work closely with policy-makers at the provincial and regional/local levels, MCHP has limited direct contact with federal politicians. Still, nationally-distributed MCHP reports, publications, conference presentations and op-eds have often stimulated interest and conversation on health and social policy topics through our media presence and other knowledge exchange activities. For example, two of our papers on Manitoba’s Healthy Baby program showing important benefits for maternal and child health were published in high-profile journals [34, 35], generating immediate commentary from other experts and resulting in invitations to present on health equity panels in Washington, D.C., San Francisco and at the Basic Income Lab at Stanford University. As well, recent work at MCHP in the area of autism generated funding from the Canadian Autism Spectrum Disorder Alliance for a national autism data strategy [36]. And MCHP has shown leadership in gaining access to social data and using it in research, allowing us to factor the social determinants of health into our analyses. The use of these additional data resources has been a model for population health data centres across Canada, inspiring other jurisdictions to begin to acquire similar databases. In this way, evidence generated by MCHP continues to play a role on the national health policy stage even without direct relationships with federal government representatives.
Besides policy-makers, another important audience for MCHP’s research is the general public. The public plays a role in influencing health and social policy, since they are voters and taxpayers with their own priorities and values, and can make their voices heard through municipal, provincial and federal channels. We engage the public in our research in various ways, including our long-standing practice of providing short lay summaries of our research (known colloquially as “4-pagers” since they are by design typically only 4 pages long), written at an accessible reading level and released alongside our technical reports, as well as creating infographics and carefully crafted social media messages. We also host a Public Engagement Group at MCHP where we invite members of the public to act in an advisory capacity. In this role, members of the public work in partnership with MCHP leadership and researchers to shape the direction of new research initiatives, discuss upcoming research projects, ensure the needs and priorities of the public are reflected in the research, and help identify possible areas of need and opportunity for public health and social policy research.
Working with media
The founding directors of MCHP understood the important role of media in reaching the public with the findings of their research [20]. When the press gives extensive coverage to MCHP reports, both government and non-government stakeholders (such as hospitals and physicians) are forced to look closely at the research and respond to it. MCHP has continued to nurture relationships with media outlets to ensure our research findings reach their intended audiences, one of which is often the public. Relationships with members of the press have led to MCHP leadership being seen as expert commentators on health policy issues by the media, which was not always well received within government, particularly when media reports were not complimentary of government policy. Social media has dramatically changed the public discourse since the early days of MCHP. The use of photos and videoclips, infographics, newsfeeds and other regular updates provides real-time information directly to a wide audience. While policy is often most relevant locally, comparative studies across jurisdictions have become more common and social media has stimulated broader discussion of findings.
Closing the loop
Policy-makers pay more attention to research findings in which they have invested time and funds. They are responsible for getting a “return” on their investments: within the public healthcare system in Canada, politicians are responsible to the electorate, civil servants answer to their political bosses, and system managers are responsible to funders. At each level of government, policy-makers invest in the issues they perceive to be most important to the system. MCHP is dependent on government departments’ continued investments to make data available and to make the data transfer processes secure. We “close the loop” by sharing relevant results with the departments who provide us their data.
MCHP is trusted with de-identified health and social data with the express purpose of using the data for research for the public good. This includes research and evaluation performed specifically at the request of government (the data provider, in most cases) and investigator-initiated research funded through the public peer review process and led by researchers at MCHP or by other researchers who request access to the data through MCHP. The informal social contract between MCHP and our data providers stipulates that we share our results with the data providers, and in this way, the resources the data providers invest in providing their data to MCHP are repaid through the process of knowledge sharing. Policy-makers across health and social departments have supported the growth of the Repository over the last 30 years because doing so directly benefits their work in the policy realm. Closing the loop by sharing our results has been crucial to the sustainability and growth of the Repository.
Discussion
The Manitoba Centre for Health Policy has had a strong relationship with the Ministry of Health in the provincial government from its very beginning. That early relationship was initially based on personal relationships with both civil servants and their political leaders, as well as on a more formal contractual relationship with baseline funding to MCHP in exchange for mutually agreed upon research deliverables. The rolling five-year contracts provided a stable fiscal environment for MCHP that supported long-term staffing and continuity despite changes in government.
Personal relationships (and the trust and respect they are built upon) require ongoing nurturing, especially during transitions in power and leadership. Building relationships at the political and the civil service levels is crucial to weather potential storms due to changes in leadership. This is especially true during political transition when mistrust can create uncertainty and challenge a longstanding partnership. In our experience, relationships with civil servants have successfully sustained the ongoing relationship through political transition.
Our experience working with policy-makers has demonstra−ted the importance of sharing our results widely. A study performed under the contract with the Ministry of Health in which we examined the relationship between the social determinants of health and primary care service quality resulted in the local Winnipeg Regional Health Authority hiring an Income Support Professional to work with low-income patients in a community clinic [37]. Sharing our findings directly with the regional service provider resulted in policy implementation with the aim of improving outcomes for a vulnerable population.
At MCHP, studies contracted by government are guided by advisory groups, which meet two or three times during the course of the study. These groups include decision-makers and policy analysts from government as well as content experts recruited by the study team. While the intent of establishing and engaging an advisory group is to ensure that the team addresses the questions of interest to the contractor, they have had additional research impacts. For example, in one study, as the research results emerged from the analyses, discussion amongst advisory group members led to changes in emergency room care processes [38, 39]. These direct and rapid changes in policy are rare, but exciting for the research team to witness.
Regional policy implementation is supported by engaging NTK Team representatives from each region. Members of the team are kept informed of new research findings at MCHP though regular meetings, providing opportunities for them to explore what the findings mean and how they could impact each region’s policy planning. Those representatives then share this information with the leadership teams on which they serve. A similar process occurs on a larger scale at our annual Evidence-to-Action workshops. Staff from each health region and from the multiple government departments that provide data to the Repository participate in these workshops. Scientists from MCHP present recent findings from government- and publicly-funded research studies and then facilitate cross-sector discussions amongst participants. In facilitated breakout groups, we help all participants understand the findings and explore their relevance to each department or region. Participants then take the new knowledge back to their workplaces and incorporate the new evidence into their day-to-day operations.
The Manitoba model for influencing policy-making using whole-population administrative data has many strengths, which together have formed the foundation of an innovative, adaptable and successful population data centre that has weathered many challenges, including changes in leadership and government priorities. Many elements of what makes MCHP’s model work are highly customizable and could be adopted by other research units across Canada and internationally. MCHP’s and other data centres’ leadership reciprocal participation on each other’s advisory boards provides an opportunity to share the strategies that have led to success and will ensure a sustainable future.
Conclusion
Policy-making is a complex endeavor subject to multiple potential influences – of which research is only one. Relationships between policy-makers and researchers are the key to influencing policy, but it takes time to establish and build the trust-based relationships that open the door to including research findings in policy decisions. Nurturing these relationships over time is a key and ongoing responsibility of health and social policy research leaders. We often hear how “the results speak for themselves”, but when it comes to presenting research evidence to policy-makers that is just not the case. Accordingly, there is now a near-ubiquitous requirement by funding agencies to include knowledge translation plans in research proposals and to ensure that public, patient, government and/or community stakeholders are engaged in the research. Researchers need to develop skills not only in sharing their results through stories and metaphors that speak to targeted audiences, but also in fostering relationships with these knowledge users. Being able to interact directly with decision-makers and policy-makers improves uptake of research findings, thereby informing policies, programs and service delivery, and benefitting individuals, communities and society as a whole.
Conflicts of interests
We have no conflicts of interest to declare.
Ethics
This article did not require the involvement of any research participants or the use of their personal health information, and therefore did not require ethical approval.
Abbreviations
MCHP | Manitoba Centre for Health Policy |
NTK Team | Need-to-Know Team |
Article Details
References
1. Smith M, Roos L, Burchill C, Turner K, Towns D, Hong S, et al. Health Services Data: Managing the Data Warehouse: 25 years of Experience at the Manitoba Centre for Health Policy. In: Sobolev B, Levy A, Goring S, editors. Data and Measures in Health Services Research. Boston, MA: Springer; 2015. p. 1–26. https://doi.org/10.1007/978-1-4899-7673-4_3-1
2. Katz A, Enns J, Smith M, Burchill C, Turner K, Towns D. Population Data Centre Profile: The Manitoba Centre for Health Policy. Int J Popul Data Sci. 2019;4(2):10. https://doi.org/10.23889/ijpds.v5i1.1131
3. Roos N, Shapiro E. Monitoring the Winnipeg Hospital System: The First Report [Internet]. Winnipeg, MB; 1994. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/BedClz.pdf
4. Roos N, Fransoo R, Bogdanovic B, Friesen D, MacWilliam L. Issues in the Management of Specialist Physician Resources for Manitoba [Internet]. Winnipeg, MB; 1997. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/SpclPlan.pdf
5. DeCoster C, Kozyrskyj A. Long-Stay Patients in Winnipeg Acute Care Hospitals [Internet]. Winnipeg, MB; 2000. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/longstay.pdf
6. Brownell M, Lix L, Ekuma O, Derksen S, De Haney S, Bond R, et al. Why is the Health Status of Some Manitobans Not Improving? The Widening Gap in the Health Status of Manitobans [Internet]. Winnipeg, MB; 2003. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/hlthgap.pdf
7. Brownell M, Roos NP, Fransoo R, Guevremont A, MacWilliam L, Derksen S, et al. How do educational outcomes vary with socioeconomic status? Key findings from the Manitoba Child Health Atlas 2004. Winnipeg, Canada: Manitoba Centre for Health Policy; 2004.
8. Brownell M, Santos R, Kozyrskyj A, Roos N, Au W, Dik N, et al. Next Steps in the Provincial Evaluation of the BabyFirst Program: Measuring Early Impacts on Outcomes Assciated with Child Maltreatment [Internet]. Winnipeg, MB; 2007. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/BF_web.pdf
9. Brownell M, Chartier M, Santos R, Ekuma O, Au W, Sarkar J, et al. How are Manitoba’s children doing? Winnipeg, MB: Manitoba Centre for Health Policy; 2012.
10. Brownell M, Chartier M, Au W, MacWilliam L, Schultz J, Guenette W, et al. The educational outcomes of children in care in Manitoba [Internet]. Winnipeg, MB; 2015. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference//CIC_report_web.pdf
11. Brownell M, Nickel N, Turnbull L, Au W, Ekuma O, MacWilliam L, et al. The Overlap Between the Child Welfare and Youth Criminal Justice Systems: Documenting “Cross-Over Kids” in Manitoba [Internet]. Winnipeg, MB; 2020. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/MCHP_JustCare_Report_web.pdf
12. Finlayson G, Smith M, Burchill C, Towns D, Peeler W, Soodeen R-A, et al. Social Housing in Manitoba. Part I: Manitoba Social Housing Data [Internet]. Winnipeg, MB; 2013. Available from: http://mchp-appserv.cpe.umanitoba.ca/deliverablesList.html
Smith M, Finlayson G, Martens P, Dunn J, Prior H, Taylor C, et al. Social Housing in Manitoba. Part II: Social Housing and Health in Manitoba: A First Look [Internet]. Winnipeg, MB; 2013. Available from: http://mchp-appserv.cpe.umanitoba.ca/deliverablesList.html
14. Roos L, Jarmasz J, Martens P, Katz A, Fransoo R, Soodeen, R-A, Smith M, et al. Health Services Information: From Data to Policy Impact (25 Years of Health Services and Population Health Research at the Manitoba Centre for Health Policy). In: Sobolev B, Levy A, Goring S, editors. Data and Measures in Health Services Research [Internet]. Boston, MA: Springer; 2015. Available from: https://doi.org/10.1007/978-1-4899-7673-4_9-1
15. Roos LL, Menec V, Currie RJ. Policy analysis in an information-rich environment. Soc Sci Med. 2004 Jun;58(11):2231–41. https://doi.org/10.1016/j.socscimed.2003.08.008
16. Roos LL, Nicol JP. A research registry: uses, development, and accuracy. J Clin Epidemiol. 1999;52(1):39–47. https://doi.org/10.1016/s0895-4356(98)00126-7
17. Roos LL, Brownell M, Lix L, Roos NP, Walld R, MacWilliam L. From health research to social research: Privacy, methods, approaches. Soc Sci Med. 2008;66(1):117–29. https://doi.org/10.1016/j.socscimed.2007.08.017
18. Smith M, Lix LM, Azimaee M, Enns JE, Orr J, Hong S, et al. Assessing the quality of administrative data for research: A framework from the Manitoba Centre for Health Policy. J Am Med Inf Assoc [Internet]. 2017;ocx078. Available from: https://doi.org/10.1093/jamia/ocx078
19. Roos LL, Gupta S, Soodeen R, Jebamani L. Data quality in an information-rich environment: Canada as an example. Can J Aging. 2005;24(Suppl 1):153–70. https://doi.org/10.1353/cja.2005.0055
20. Roos NP, Roos LL, Freemantle J. Administrative Data and the Manitoba Centre for Health Policy: Some Reflections. Healthc Policy [Internet]. 2011 Jan [cited 2021 Sep 21];6(Sp):16. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5319570/
21. Harron K, Dibben C, Boyd J, Hjern A, Azimaee M, Barreto M, et al. Challenges in administrative data linkage for research. Big Data Soc. 2017;4(2). https://doi.org/10.1177/2053951717745678
22. Katz A, Enns J, Wong S, Williamson T, Singer A, McGrail K, et al. Challenges Associated with Cross-Jurisdictional Analyses using Administrative Health Data and Primary Care Electronic Medical Records in Canada. Int J Popul Data Sci. 2018;3(3). https://doi.org/10.23889/ijpds.v3i3.437
23. Nickel NC, Turnbull L, Wall-Wieler E, Au W, Ekuma O, MacWilliam L, et al. Overlap between child protection services and the youth justice system: protocol for a retrospective population-based cohort study using linked administrative data in Manitoba, Canada. BMJ Open. 2020;10(7):e034895. https://doi.org/10.1136/bmjopen-2019-034895
24. Nickel NC, Clark W, Phillips-Beck W, Sanguins J, Enns JE, Lavoie JG, et al. Diagnostic testing and vaccination for COVID-19 among First Nations, Metis and Inuit in Manitoba, Canada: protocol for a nations-based cohort study using linked administrative data. BMJ Open [Internet]. 2021 Sep 1 [cited 2021 Sep 21];11(9):e052936. Available from: https://bmjopen.bmj.com/content/11/9/e052936. https://doi.org/10.1136/bmjopen-2021-052936
25. Brownell MD, Nickel NC, Enns JE, Chartier M, Campbell R, Phillips-Beck W, et al. Association between home visiting interventions and First Nations families’ health and social outcomes in Manitoba, Canada: protocol for a study of linked population-based administrative data. BMJ Open. 2017 Oct;7(10):e017626. https://doi.org/10.1136/bmjopen-2017-017626
26. Demeter SJ. Helping Health Care Providers and Clinical Scientists Understand Apparently Irrational Policy Decisions. Cureus [Internet]. 2016 Dec 21 [cited 2021 Sep 21];8(12). Available from: https://www.cureus.com/articles/5924-helping-health-care-providers-and-clinical-scientists-understand-apparently-irrational-policy-decisions
27. Martens PJ, Roos NP. When Health Services Researchers and Policy Makers Interact: Tales from the Tectonic Plates. Healthc Policy [Internet]. 2005 Sep 15 [cited 2021 Sep 21];1(1):72. Available from: /pmc/articles/PMC2585237/
28. Bowen S, Erickson T, Martens P, Crockett S. More than “using research”: the real challenges in promoting evidence-informed decision-making. Heal Policy. 2009;4(3):87–102.
29. Bowen S, Martens P, The Need to Know Team. Demystifying knowledge translation: learning from the community. J Heal Serv Res Policy. 2005;10(4):203–11. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2653695/
30. The First Nations Information Governance Centre. Ownership, Control, Access and Possession (OCAP): The Path to First Nations Information Governance [Internet]. Ottawa: The First Nations Information Governance Centre; 2014. Available from: http://fnigc.ca/sites/default/files/docs/ocap_path_to_fn_information_governance_en_final.pdf
31. First Nations Metis and Inuit Health Research Strategic Planning Committee. Framework for Research Engagement with First Nations, Metis, and Inuit Peoples [Internet]. Winnipeg, MB; 2013. Available from: https://umanitoba.ca/faculties/health_sciences/medicine/media/UofM_Framework_Report_web.pdf
32. Marchessault G. The Manitoba Centre for Health Policy: A Case Study. Healthc Policy. 2011;6(Spec Issue):29–43. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5319571/
33. ICES Ontario. Pat Martens (The Little Black Dress) [Internet]. 2012 [cited 2021 Sep 22]. Available from: https://www.youtube.com/watch?v=RKpyxFazSeQ
34. Brownell MD, Chartier MJ, Nickel NC, Chateau D, Martens PJ, Sarkar J, et al. Unconditional Prenatal Income Supplement and Birth Outcomes. Pediatrics. 2016;137(6):e20152992.
35. Brownell M, Nickel N, Chartier M, Enns J, Chateau D, Sarkar J, et al. An unconditional prenatal income supplement reduces population inequities in birth outcomes. Health Aff. 2018 Feb 1;37(3):447–55. https://doi.org/10.1377/hlthaff.2017.1290
36. Canadian Autism Spectrum Disorder Alliance. Roadmap to a National Autism Strategy [Internet]. 2020. Available from: https://www.casda.ca/wp-content/uploads/2020/03/Roadmap-to-a-National-Autism-Strategy_CASDA_March-2020.pdf
37. Katz A, Chateau D, Enns JE, Valdivia J, Taylor C, Walld R, et al. Association of the social determinants of health with quality of primary care. Ann Fam Med. 2018 May 1;16(3):217–24. https://doi.org/10.1370/afm.2236
38. Doupe MB, Chateau D, Chochinov A, Weber E, Enns JE, Derksen S, et al. Comparing the Effect of Throughput and Output Factors on Emergency Department Crowding: A Retrospective Observational Cohort Study. Ann Emerg Med. 2018 Oct 1;72(4):410–9. https://doi.org/10.1016/j.annemergmed.2018.04.001
39. Doupe M, Chateau D, Derksen S, Sarkar J, Lobato de Faria R, Strome T, et al. Factors Affecting Emergency Department Waiting Room Times in Winnipeg [Internet]. Winnipeg, MB; 2017. Available from: http://mchp-appserv.cpe.umanitoba.ca/reference/ER2011_report_Web.pdf