How to analyze and link patient experience surveys with administrative data to drive health service improvement -- examples from Alberta, Canada

Abstract

The ability of hospitals and health systems to learn from those who use its services (i.e., patients and families) is crucial for quality improvement and the delivery of high-quality patient-centered care. To this end, many hospitals and health systems regularly collect survey data from patients and their families, and are engaged in activities to publicly report the results. Despite this, there has been limited research into the experiences of patients and families, and how to improve them. Since 2015, our research team has conducted a variety of studies which have explored patient experience survey data, in isolation, and in linkages with routinely-captured administrative data sets across Alberta; a Canadian province of 4.4 million residents. Via secondary analyses, these studies have shed light upon the drivers of inpatient experience, the specific aspects of care which are most correlated with one's overall experiences, and the association of elements of the patient experience with other measures, such as patient safety indicators and unplanned hospital readmissions. The aim of this paper is to provide an overview of the methods we have used, including further details about the data sets and linkage protocol. The main findings from these papers have been presented for readers and those who wish to conduct their own work in this area.

Introduction

In 2001, the Institute of Medicine (IOM), presented six key aims for improving healthcare; one of which was that care should be patient-centered [1]. At that time, the IOM defined patient-centered care (PCC) as “providing care that is respectful of and responsive to individual patient preferences, needs, and values, and ensuring that patient values guide all clinical decisions” [1]. From this, PCC places the patient at the centre of their care, ensuring that care decisions and treatments are the result of shared decision making between the patient and their healthcare provider/team. In recent years, the concept of PCC has expanded to recognize the role of family members and caregivers, as they also may play an active role in the patient’s care [2].

To measure and assess how well hospitals and health care systems are doing in providing PCC, it is important to consider and document the perspectives of patients. Experience surveys are widely used to obtain feedback from patients and families about the healthcare services that they receive. The insights from such survey data may be used to shed light upon leading practices and serve to design and improve programs and services which champion PCC. In the inpatient setting, validated surveys such as the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) [3, 4], the Canadian Patient Experiences Survey – Inpatient Care (CPES-IC) [5], and the Picker Patient Experiences Questionnaire (PPEQ) [6] have been used to assess the patient experiences with hospital care.

The importance of the patient experience to assess quality of care is now widely recognized. Since 1998, patient experience has been one of the three key aims of the Institute for Healthcare Improvement’s Triple Aim Framework [7]. From this, a successful hospital or health system is one which strives to attain the best experience for their patients. In recent years, many hospitals and health systems have evolved to now include patient experience measures as part of their key performance indicators and/or public reporting. In the United States, hospitals are mandated to publicly report their patient experience results [8]. In our province, Alberta Health Services (AHS) reports the overall experience rating of their hospital patients on a quarterly basis [9]. Beyond public reporting, there are many opportunities for deeper analysis of patient experience data to inform the delivery of high-quality care.

The goal of this manuscript is to provide readers with more information about how our research team uses stand-alone and linked patient experience survey data to assess quality of care in Alberta, Canada. We provide details about the data sources, linkage protocol, analysis methods, and key findings from our ongoing program of research.

Methods

Data sources

Survey data

AHS is the sole provider of inpatient (i.e., hospital) care for Alberta’s 4.4 million residents [10]. The health authority has conducted surveys in the inpatient setting for over a decade. Amongst adults, the Canadian Patient Experiences Survey – Inpatient Care (CPES-IC) [11], an instrument developed by the Canadian Institute for Health Information (CIHI), has been in use since April 2014. The experiences of parents and guardians of hospitalized children has been captured using the Alberta Pediatric Inpatient Experiences Survey (APIES), a survey adapted from the Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child-HCAHPS) [12] since October 2015. The CPES-IC and APIES are validated instruments which are administered via telephone. The CPES-IC is captured from 93 hospitals across the province, and the APIES is captured from 13 (2 stand-alone children’s hospitals and 11 other sites which treat large volumes of pediatric patients). Each year, approximately 25,000 adult, and 2,500 pediatric (child and newborn) inpatient surveys are obtained. Potential respondents are selected randomly and contacted between 2- and 42-days following discharge from hospital. To maximize the chance for each respondent to complete the survey and to obtain a diverse sample, dialing occurs on varying days (e.g., weekdays, weekends), at varying times (e.g., mornings, during the day, evenings). Calls are completed from 9AM to 9PM on weekdays, and from 10AM to 4PM on weekends.

The CPES-IC and APIES consist of 56 and 66 questions, respectively. Both surveys ask respondents to evaluate multiple aspects of care, including admission to the hospital, communication with nurses and doctors, pain control and medications, the physical hospital environment, information exchange, and discharge planning. Respondents are also asked to provide an overall rating of care on a scale from 0 (worst) to 10 (best), and whether they would recommend the hospital to family members and/or friends. Responses to survey questions are Likert scales (e.g., always, usually, sometimes, never). Each of the surveys contains a demographic section, in which respondents are asked to provide their age and highest level of educational attainment. On the CPES-IC, respondents provide their self-reported levels of physical and mental health, while on the APIES, adults provide a rating of health status of the child.

The adult survey contains a section on concerns with care. Respondents are asked if they had any concern(s) about their care. If a respondent replies “yes”, they are then asked an open-ended question about the nature of their concern. They are then asked what they did about their concern, which could include discussion with the healthcare provider/team, informing the organization via email, phone call, or letter. Those who respond with any of these options are asked about their satisfaction regarding how their concern was received, addressed, and the overall handling of their concern. Both surveys (CPES-IC and APIES) conclude with an open-ended question which asks respondents if there is anything additional that they would like to share about their hospital experience.

Administrative data

AHS also houses a variety of clinical data which may provide relevant information that is not contained in the surveys. Our research has utilized data from two of these large administrative data sets. In the inpatient setting, the Discharge Abstract Database (DAD) [13] contains records from all inpatient discharges across the province. Records include demographic and clinical variables such as patient age and sex, visit type (urgent, elective, etc.), length of stay, and discharge disposition. Each record also includes up to 25 diagnoses, coded according to the International Classification of Diseases, Canadian edition, 10^th version (ICD-10-CA) [14], and up to 20 procedure codes, coded according to the Canadian Classification of Interventions (CCI) [15].

Data from emergency departments is housed in the National Ambulatory Care Reporting System (NACRS) [16]. In Alberta, this includes all data from stand-alone emergency departments, urgent care centres, day surgeries, and some pre-booked clinic visits. Like with the DAD, the NACRS database includes demographic and clinical data. Demographic information includes sex and age, while clinical data includes the visit type (emergent/urgent, other), disposition, length of stay, up to 10 diagnoses (also coded via ICD-10-CA), and up to 8 procedures (also coded via CCI). In the case of emergency department and urgent care centre visits, the level of patient acuity at registration is documented according to the Canadian Triage Acuity Score (CTAS) [17].

Record linkage

All of the data sources (surveys, administrative) used in our research program are housed by AHS. Surveys are linked with each corresponding index inpatient record, using exact matches of personal health number, five-digit hospital code and discharge date. This ensures that each survey is linked with the correct index hospital admission, as some patients may have multiple hospital visits. To explore previous and subsequent acute care utilization (e.g., other hospitalizations and emergency department visits), each survey is then linked to all pertinent records for each patient based on personal health number and visit dates.

Analysis

To date, our research team has analyzed the stand-alone survey data using a variety of methods. The first is to report the percentage of “top box” responses to each survey question. A “top box” response reflects the most positive answer choice to the question asked [18, 19]. For example, on the question regarding how often nurses explained things in a way the patient could understand, the possible responses are always, usually, sometimes, and never. The “top box” response to this question is “always”. On questions which ask respondents to provide an overall rating (e.g., overall experience, overall hospital) the “top box” response is a score of 9 or 10. This corresponds with the methods used in the “Net Promoter Score”, a widely-used metric which measures consumer loyalty [20].

The second method is to classify the survey questions into domains/composites, as per HCAHPS [21] and Child-HCAHPS [22] methodology, and to report the “top box” score for each domain/composite. As an example, “communication with nurses” on the adult survey is comprised of three questions: how often nurses a) treated you with courtesy and respect, b) listened carefully to you, and c) explained things in a way you could understand. The average of the three top box percentages for these questions is reported as the domain/composite score.

We have also examined the correlation between survey questions/domains and the overall rating of care. This is extremely important from a quality improvement perspective, as it allows us to determine which aspects of care, when done well, should have the most impact upon one’s overall experience with care. Aspects relating to communication with nurses are typically the most correlated with overall rating [23–25]. From this knowledge, efforts to improve or maintain optimal nurse communication should ensure a high rating of care from patients.

Our team has also explored the open-ended responses that patients have provided regarding their concerns with care. The analysis of open-ended comments has typically been done in two ways. First, we have employed traditional qualitative analyses, where two reviewers have classified concerns data into a list of themes [26]. Percent agreement between the reviewers was measured, and disagreements were resolved during a consensus meeting. Recognizing that this is a very labour-intensive process, our team has explored the use of natural language processing to analyze patient concerns data [27]. This second method is quite advantageous as it can be done in real-time. As such, the duration between data capture and reporting to administration and members of the care team is much shorter. Further, this approach may uncover emerging themes in the data – ones which are not defined a priori. The capabilities and challenges of using natural language processing to examine open-ended patient comments have been recently presented in a report from the RAND Corporation [28].

The majority of our work to date has been completed using linkages between the survey data and administrative (e.g., inpatient, emergency department) datasets. Linkage with these administrative sources, as described in the process above, provides a wealth of clinical information that is not available in the survey alone (e.g., most responsible diagnosis, procedures performed, number of medical comorbidities, etc.). The linked data allows us to conduct stratified analyses (e.g., based on age, sex, length of stay, etc.), to examine the experiences of pre-defined clinical cohorts (e.g., by diagnosis, surgical procedure, time period), and to examine the potential associations of survey data with other outcomes (e.g., readmissions, emergency department visits, patient safety indicators) in regression analyses. Determining whether patient experience varies according to clinical and/or demographic factors is necessary for conducting proper case-mix adjustment and reporting of results [29–31].

Results

From April 2014 to September 2020, over 155,000 CPES-IC surveys had been completed by adults following their discharge from hospital. From October 2015 to March 2021, over 14,000 APIES surveys were completed by parents/guardians of children who were hospitalized. Over these time periods, the majority of adult (99.4%) and child (98.8%) surveys have been successfully linked with the corresponding inpatient record from the DAD, via the criteria described in the methods section. A summary of manuscripts published by our research team using this data, is provided in Table 1. For each manuscript, a brief description, the objective(s), data set(s) used, and main findings are provided. As shown in the table, and highlighted in the methods section, the majority of our work has used linkages of multiple data sources. The main findings from each of our manuscripts are summarized below.

Table 1: Published research using patient experience data.

Study description	Objective(s)	Data source(s)	Main findings
Correlation with overall experience (adult) [24]	Determine which individual patient experience questions and domains were most correlated with overall inpatient hospital experience	CPES-IC	Questions on provider coordination and nursing care were most correlated with overall experience. Hospital cleanliness, quietness, and discharge information questions showed poor correlation. Correlation with overall experience was strongest for the “communication with nurses” domain.
Correlation with overall rating (child) [25]	Determine which survey questions were most correlated with respondents’ overall rating of care.	APIES	Questions on provider coordination and nursing care were most correlated with overall experience. Quietness of hospital room and keeping families informed in the emergency room showed poor correlation. Correlation with overall experience was strongest for the “communication with nurses” domain.
Analysis of patient concerns [26]	Determine (1) the frequency of qualitative complaints and the demographic/clinical profile of patients lodging them, (2) the most frequent complaint themes and their association with overall experience scores and (3) whether overall experience scores varied based upon the complaint action taken by the patient or the degree of patient satisfaction in the handling of complaints.	CPES-IC, DAD	Most frequent complaint themes were nursing, medications, and food. Increased odds of having a complaint were associated with younger age, being born in Canada and having no documented medical comorbidities. Protective factors were male gender, lower education level, urgent hospital admission, lower resource intensity and length of stay <3 days.
Categorizing patient concerns using natural language [27] processing	Report the results of using a semi-automated, computational topic-modelling approach to analyse a corpus of patient feedback.	Patient concerns database	The LDA model produced 40 topics which, following manual interpretation by researchers, were reduced to 28 coherent topics. The most frequent topics identified were communication issues causing delays, community care for elderly patients, interactions with nurses, and emergency department care. Many patient concerns were categorized into multiple topics. Some were more specific versions of categories from the existing framework (e.g., communication issues causing delays), while others were novel (e.g., smoking in inappropriate settings).
Survey sample representativeness [32]	Compare selected demographic and clinical attributes of survey respondents to those of all eligible inpatient discharges over the same time period.	CPES-IC, DAD	The survey sampling strategy resulted in a sample that was, in most cases, representative of the general inpatient population in our jurisdiction of approximately 4 million residents.
Drivers of overall experience (adult) [33]	Identify factors associated with patients’ overall rating of inpatient hospital care.	CPES-IC, DAD	Overall hospital experience of 0–9 was associated with younger age, male gender, higher level of education, being born in Canada, urgent admission, not having a family practitioner as the most responsible provider service, and not being discharged home. A length of stay of less than 3 days was protective.
Drivers of overall experience (child) [34]	Determine the patient and respondent factors associated with pediatric inpatient experiences.	APIES, DAD	Case-mix characteristics were unevenly distributed between general and pediatric hospitals. Older respondents, healthier patients, and treatment at pediatric facilities had increased odds of providing most-positive ratings. Increased respondent education was associated with decreased odds of providing most-positive ratings.
Family experiences with pediatric inpatient care [35]	Describe the use of the Child HCAHPS survey, and to present preliminary results.	APIES, DAD	Mean top box scores ranged from 41.5% (“preventing mistakes and helping you report concerns”) to 95.8% (“keeping you informed about your child’s care in the emergency department”). Stand-alone pediatric hospitals tended to outperform the adult ones, particularly in global ratings of care, parental involvement in decision-making, and communication between parents and providers.
Adult experiences with hospitalization during the COVID-19 pandemic [36]	(a) Report on the experiences, (b) Compare with historical results, and (c) Assess for potential monthly differences of patients hospitalized in the early months of COVID-19.	CPES-IC, DAD	Those hospitalized during COVID-19 had higher odds of “top box” ratings on 17 of 39 questions examined, and lower odds on 2 questions (information about the admission process, inclusion of family/friends in care decisions). The remaining 20 questions showed no difference between the two cohorts.
Using design-thinking to investigate and improve patient experience [37]	Capture a comprehensive and nuanced understanding of the patient experience on an inpatient medical teaching unit in order to identify key deficiencies and unmet needs.	CPES-IC	We identified several key deficiencies including call bell response times, noise levels at night, pain control, education about medication side effects, communication between healthcare team members, and how well healthcare team members remain up to date about patient care.
Patient experiences with coronary artery bypass grafting and valve replacement [38]	Examine the comprehensive experience of patients who have undergone CABG and/or valve replacement.	CPES-IC, DAD	Top performing questions pertained to having a discussion about help needed after discharge and receiving written discharge information. Lack of quietness of the hospital environment at night and lack of staff sufficiently describing side effects of new medications were identified as potential areas for improvement.
Sex differences in the experiences of patients hospitalized due to ischaemic heart disease [39]	Examine the comprehensive experience of patients hospitalized due to ischemic heart disease, according to sex.	CPES-IC, DAD	Women had lower top-box percentages on 26 of 37 questions. Similar results were obtained for the adjusted odds of reporting a top-box response. Women did not have a higher percentage of top-box responses on any of the questions studied.
Experience of parents of hospitalized children living with medical complexity (CMC) [40]	Examine the comprehensive inpatient experience of CMC, and compare the results with all other respondents at two academic pediatric hospitals.	APIES, DAD	Among CMC, the highest-rated measures pertained to being kept informed while in the emergency department, a willingness to recommend the hospital, and parents having a clear understanding of their role in their child’s care. The lowest-rated measures pertained to preventing mistakes and reporting concerns and the quietness of the hospital room at night. Compared with others, parents of CMC reported lower raw results on 20 of the 28 measures.
Association of overall experience with patient safety indicators [41]	Document the association of PSIs and patient experience scores, and to determine risk-adjusted odds of high experience scores versus PSI presence.	CPES-IC, DAD	Overall, physician and nurse care was rated high by 61.9%, 73.7% and 66.2% of respondents. 1085 patients (4.3%) had a documented PSI. Risk-adjusted models showed patients with PSIs had decreased odds of having high overall, physician and nurse ratings.
Association of patient involvement in care decisions and discharge instructions with unplanned readmissions [42]	Examine the potential associations between patient-reported aspects of communication and discharge care with unplanned readmissions up to one year post-discharge.	CPES-IC, DAD	Patients who felt they were not involved in care decisions were more likely to be readmitted, as were patients who reported not receiving written information about signs and symptoms to watch out for post-discharge.
Association with patient-reports of night noise in hospitals with unplanned readmissions among older adults [43]	Examine the relationship between patient-reported hospital quietness at night, via a validated survey, and unplanned hospital readmissions among hospitalized seniors (ages 65 and older).	CPES-IC, DAD	Patients not reporting “always” to the survey question regarding hospital quietness at night had slightly greater odds of readmission within 30 and 90 days.
Association of Care Transitions Measure-3 with post-discharge outcomes [44]	To explore whether CTM-3 scores are a suitable proxy for quality of transitional care.	CPES-IC, DAD	CTM-3 scores were not significantly associated with outcomes at 30 days. Although CTM-3 scores were significantly lower in patients who subsequently had ED visit/readmission at 3 months and 12 months, the magnitude of risk was small: for every 10 point decrease in the CTM-3 score, the risk of ED visit/readmission was 2.6% higher at 3 months and 4.0% higher at 12 months.
Association of proxy-rated pediatric health status with unplanned health services utilization [45]	Investigate the association of family-rated health status (FRH) in pediatric care with administrative indicators, patient and respondent features, and unplanned health services use.	APIES, DAD, NACRS	Automated models of unplanned services use included FRH as a feature, and poor ratings of health were associated with increased odds of emergency department visits and readmission.
Accuracy of recall of interaction with a hospital pharmacist [46]	Determine accuracy of patients’ recollection of meeting with a pharmacist at two acute care teaching hospitals.	CPES-IC, DAD, Patient charts	Of the 196 respondents who reported not speaking with a pharmacist, 73 (37.2%) had documentation present. Compared with patient charts, sensitivity and specificity were 49.3% and 49.8%, respectively. Positive and negative predictive values were 36.4% and 62.8%, respectively.

As non-response bias can be a significant limitation of surveys, a necessary first project was to compare the demographic and clinical characteristics of those who completed a survey, versus those who were otherwise eligible but did not (e.g., those who were not randomly selected or refused). To do so, we reported on the analysis of one year’s worth of linked data (adult surveys plus inpatient records). We observed that the patient profile of respondents was similar to that of the other inpatients across our province. Respondents and non-respondents were similar in terms of age, sex, admission type (urgent vs. elective), and number of medical comorbidities. We did find, however, that survey respondents tended to remain in hospital for a shorter period of time, require ICU care less frequently, and were more likely to be discharged home (as opposed to being transferred to another acute care hospital, or long-term care facility) [32].

To examine the correlation between individual survey questions and domains with the overall rating of care, we performed analyses of the survey data in isolation (without linkage to other data sets). Among adults, survey questions on provider coordination and nursing care were most correlated with overall experience, while the domain pertaining to communication with nurses showed the highest correlation [24]. Similar results were observed among parents/guardians of hospitalized children [25].

With respect to the drivers of overall experience among adults, we observed, using survey data linked with inpatient records, that younger adults, males, those with higher levels of education, those born in Canada, patients admitted to hospital on an urgent basis, those not having a primary care provider as the attending physician, and not being discharged home reported lower ratings of overall experience [33]. Among parents of hospitalized children, we observed similar results; that younger respondents, those with poorer parent-reported child health, higher levels of education, and children not treated in stand-alone pediatric hospitals reported lower odds of reporting a “top box” overall rating [34].

When examining all questions on both the adult and child experience surveys, we have observed that respondents tend to have high ratings of their overall care. We have also observed, however, that improvements could be made with respect to information sharing, particularly with respect to informing patients about the potential side effects of medications that patients had not taken before, informing parents on how they could report concerns, and providing written information to patients about signs and symptoms to watch out for after their discharge from hospital [35, 36]. Another study, conducted on a single medical teaching unit, highlighted key patient-reported deficiencies including call bell response times, noise levels at night, pain control, education about medication side effects, communication between healthcare team members, and how well healthcare team members remain up to date about patient care [37].

Many of our projects have used comparative analyses to examine differences in patient experience among specific clinical or demographic cohorts. We have done this by selecting specific diagnoses and/or procedures from the inpatient data. Among patients who underwent cardiac surgery, we reported that noisiness of the hospital environment and the lack of information provided about potential side effects of new medications were areas which could be improved [38]. In another report, we showed that women who were hospitalized due to ischaemic heart disease reported a worse experience on over half of the 37 survey questions examined, when compared with their male counterparts [39]. Similarly, we showed that parents of hospitalized children living with medical complexity reported a poorer experience on over 70% of the survey questions, when compared with parents of other hospitalized children [40]. Recently, we showed that adults who were hospitalized during the early months of the COVID-19 pandemic reported experiences that were in many aspects, better that those who were hospitalized during the previous year (pre-pandemic) [36].

Our approach has also allowed us to use the linked data to examine the potential associations between patient experience and other health system measures and outcomes. We have shown that patients with one or more documented patient safety indicators tend to report lower overall hospital, doctor, and nurse ratings [41]. We have also conducted a variety of studies which have explored the relationships between aspects of patient experience and subsequent acute care utilization (e.g., unplanned readmissions, return visits to an emergency department). In one study, we found that a lack of patient involvement in their care decisions and not receiving written discharge instructions were associated with unplanned readmission to hospital [42]. Among older adults, we observed that those who reported night noise in their hospital environment had higher odds of being readmitted to hospital [43]. In another study among adults, we explored use of the 3-item Care Transitions Measure. We observed that the measure, which is a scale constructed from responses to three survey questions, was only weakly associated with future unplanned readmissions and emergency department visits [44]. Among children, we observed that those with poorer family-rated health status had higher odds of subsequent emergency department visits and unplanned readmissions to hospital [45]. This finding is an important one, which may be used in future case-mix adjustment of the data.

Our research has also served to examine the validity of certain survey questions. An example of this are questions on the adult survey pertaining to care delivered by a pharmacist (which is done at some, but not all hospitals in our province). In a study which compared a question regarding patients meeting with a pharmacist (yes/no) with data from patient charts and pharmacist logs, it was found that patients had poor recall of meeting a pharmacist during their hospital stay [46]. This has led to changes in the way that the question is asked of respondents to ensure better quality of the underlying data.

Finally, our team has produced two manuscripts from open-ended patient data. In the first, we examined one year of patient concerns data, as captured on the adult survey. Approximately one in five respondents reported at least one concern stemming from their hospital care. The most frequent concerns pertained to nursing care, medications, and food. We also observed that younger age, being born in Canada and having no documented medical comorbidities were associated with increased odds of having a concern, while male gender, lower education level, urgent hospital admission, lower resource intensity, and shorter length of stay were protective factors [26]. In the second manuscript, we used natural language processing to produce 28 coherent topics from a patient concerns database in AHS (not limited to acute care). The most frequent topics identified were communication issues causing delays, community care for elderly patients, interactions with nurses, and emergency department care. Many patient concerns were categorized into multiple topics. Some were more specific versions of categories from the existing framework (e.g., communication issues causing delays), while others were novel (e.g., smoking in inappropriate settings) [27].

Discussion

We are currently amid an important paradigm shift, one that recognizes the leading role that patients and families play in the creation of person-centred learning health systems. Patient and family reports of their experiences with care may provide unique information regarding the quality and effectiveness of healthcare, complimenting the perspectives of providers and administrators [47–50]. The collection and use of patient experience data has become an emerging priority for hospitals and health systems alike. This data, in isolation, or in combination with other data sources, may be used to highlight leading practices or to design interventions to improve elements of care which matter most to patients. As outlined by Friedman’s framework of learning health systems [51], a successful person-centred learning health system is one that “learns” from patients and their experiences with healthcare. It does this by addressing single incidents or sets of activities (“doing things right”), learning from its own mistakes and sense-making (“how to do the right things”) and from critical analysis of how health systems give rise to actions [51]. However, despite the vast efforts to collect patient experience data, the translation of this data into knowledge and improvement actions remains low in many areas and jurisdictions. As an established research team with expertise in this area, our aim with this manuscript was to highlight our approach for others who wish to explore and learn from their own data.

Prior to collecting any data from patients and families, however, it is extremely important that the underlying data is trustworthy and of high quality. This can be achieved by using a valid and reliable instrument, and by having a sound sampling strategy and protocol. The CPES-IC and APIES instruments are valid and reliable instruments which are used for assessing the experiences of hospitalized Canadians. The rigorous methodology for each includes a standard script, responses to frequently asked questions, and quality assurance checks, to ensure that all surveys are administered in a standardized fashion. Prospective respondents were randomly selected and provided with multiple opportunities to respond at varying times and days throughout the week. Despite these efforts, there is always a potential for non-response bias in survey research. To assess the efficacy of the sampling strategy and survey protocol, our research group conducted a study to assess this in the adult population. We observed that the patient profile of respondents was similar to that of other inpatients across our province, demonstrating the generalizability of the underlying survey data/results [32]. Given that AHS provides all inpatient hospital care across Alberta, this is an important finding.

There are many pros to linking surveys with administrative data. Linkage provides great opportunities to create richer data sets which incorporate patient-reported feedback with clinical and demographic elements – ones which may not always be present in a stand-alone survey. In our examples, using the DAD and NACRS, we were able to access structured, high-quality administrative data which allowed us to explore the experiences of patients across different demographics, conditions, time periods, and hospitals. The incorporation of administrative data may also reduce the burden placed on respondents. Elements pertaining to length of stay, readmission, clinical diagnoses and procedures were obtained from administrative data, rather than asking respondents. Finally, in being able to evaluate patient experiences alongside clinical data, our team was able to explore the potential relationships between various elements of the patient experience (e.g., discharge planning and communication) and other health system measures (e.g., unplanned readmissions). This may serve to demonstrate the clinical importance and relevance of capturing the patient voice through surveys of their healthcare experiences.

With respect to our linkage protocol, we were able to successfully link over 98% of completed surveys with their corresponding inpatient record. We have been very fortunate to work in partnership with AHS. Given that AHS uses administrative data to create the survey sample (list of patients who are eligible to complete the survey), conducts the survey themselves, and generates the inpatient and emergency department records, all of the data is contained within the same data warehouse. This ensures high quality data, while maximizing the potential for linkage between the surveys and administrative sources. We recognize that others looking to do this work may have additional challenges. Surveys and administrative data may come from different organizations and disparate databases, which may hinder linkage potential.

Of course, there are notable limitations of surveys and our underlying research which cannot be discounted. In Alberta, the CPES-IC and APIES surveys are only administered by telephone. Therefore, our findings may not be applicable to other formats, such as pen and paper, e-mail, or interactive voice response. Secondly, although the 42-day recall period used in the surveys is quite short, there is potential for recall bias among respondents. Additionally, the CPES-IC and APIES are conducted only in English in our jurisdiction, so work is needed to translate the surveys into other languages in order to capture the experiences of non-English speaking Albertans. Finally, although linkage provides opportunities for richer analysis via integration of patient-reported and clinical data, there are many data points which are not routinely captured or are done so in a non-uniform fashion. For example, data on social determinants of health could allow researchers to determine whether patient experiences vary as a product of these.

The future of patient experience data and their use for improving healthcare is bright. Many hospitals and health systems recognize the importance of harnessing patient feedback about the services they receive, and how these patient reports may be used to improve care for others. Additionally, the effective delivery of high-quality, patient-centered care is contingent on learning about the needs and values of patients, in order to align care in a personalized fashion.

Conclusions

We have provided an overview of our research program which involves the examination of patient experience data in isolation, and in combination with various administrative datasets. We discuss the methods used for linkage, analysis, and reporting of results. While patient experience data is routinely captured in several jurisdictions, there are few peer-reviewed studies which report on the insights which may be derived from patient-reported experience measures. We hope that this paper will inspire others to take a deeper dive into their own patient experience data. Patients and families play a crucial role in healthcare delivery.

Ethical approval

All necessary data for our research activities are provided by AHS, as per terms and conditions set forth in research agreements with our team. Ethical approval for all of our studies has been obtained from the University of Calgary Conjoint Health Research Ethics Board. Given the retrospective nature of our research activities, and that obtaining informed consent from those who previously completed a survey would be impractical, or in some cases, impossible, waivers of consent have been granted by the ethics board.

Conflict of interest statement

All authors have no conflicts of interest to declare.

Acknowledgements

Our research team would like to thank patients and parents for providing their feedback about their healthcare services by completing a survey. We also wish to thank Alberta Health Services for their ongoing collaboration, and their commitment to improving the experiences for their patients and families.

References

1. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.

2. NEJM Catalyst. What is Patient-Centered Care? [Internet]. Available from: https://catalyst.nejm.org/doi/full/10.1056/CAT.17.0559.

3. Giordano LA, Elliott MN, Goldstein E, Lehrman WG, Spencer PA. Development, implementation, and public reporting of the HCAHPS survey. Med Care Res Rev. 2010;67(1):27–37. 10.1177/1077558709341065

   https://doi.org/10.1177/1077558709341065

4. Goldstein E, Farquar M, Crofton C, Darby C, Garfinkel S. Measuring hospital care from the patients’ perspective: an overview of the CAHPS hospital survey development process. Health Serv Res. 2005;40(6 Pt 2):1977–95. 10.1111/j.1475-6773.2005.00477.x

   https://doi.org/10.1111/j.1475-6773.2005.00477.x

5. Hadibhai S, Lacroix J, Leeb K. Developing the first pan-Canadian acute care patient experiences survey. PXJ. 2018;5. 10.35680/2372-0247.1227

   https://doi.org/10.35680/2372-0247.1227

6. Jenkinson C, Coulter A, Bruster S. The Picker Patient Experience Questionnaire: development and validation using data from inpatient surveys in five countries. Intl J Qual Health Care. 2002;14(5):353–8. 10.1093/intqhc/14.5.353

   https://doi.org/10.1093/intqhc/14.5.353

7. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Affairs (Millwood). 2008;27(3):759–69. 10.1377/hlthaff.27.3.759

   https://doi.org/10.1377/hlthaff.27.3.759

8. Centers for Medicare and Medicaid Services. Hospital Compare: Survey of Patients’ Experiences (HCAHPS) [Internet]. Available from: https://www.medicare.gov/hospitalcompare/Data/Overview.html.

9. Alberta Health Services. Annual Report 2019-20 [Internet]. Available from: https://www.albertahealthservices.ca/assets/about/publications/2019-20-annual-report-web-version.pdf.

10. Alberta Health Services. About AHS [Internet]. Available from: https://www.albertahealthservices.ca/about/about.aspx.

11. Canadian Institute for Health Information. About the Canadian Patient Experiences Survey on Inpatient Care [Internet]. Available from: https://www.cihi.ca/en/patient-experience/about-the-canadian-patient-experiences-survey-on-inpatient-care.

12. Toomey SL, Zaslavsky AM, Elliott MN, Gallagher PM, Fowler FJ Jr, Klein DJ, et al. The development of a pediatric inpatient experience of care measure: Child HCAHPS®. Pediatrics. 2015;136(2):360–9. 10.1542/peds.2015-0966

    https://doi.org/10.1542/peds.2015-0966

13. Canadian Institute for Health Information. Discharge Abstract Database (DAD) Metadata [Internet]. Available from: https://www.cihi.ca/en/discharge-abstract-database-metadata.

14. International Statistical Classification of Diseases and Related Health Problems: Tenth Revision, Canada. Ottawa, ON: Volume One – Tabular List; 2018.
15. Canadian Classification of Health Interventions (CCI) – Tabular List. Ottawa, ON; 2012.
16. NACRS Data Elements 2021-2022. Ottawa, ON; 2021.

17. Beveridge R, Clarke B, Janes L, Savage N, Thompson J, Dodd G, et al. Canadian emergency department triage and acuity scale: implementation guidelines. Can J Emerg Med. 1999;1(Suppl):S2–S28.

18. Centers for Medicare and Medicaid Services. A Note About HCAHPS “Boxes” [Internet]. Available from: https://hcahpsonline.org/en/summary-analyses/#NoteAboutBoxes.

19. Centers for Medicare and Medicaid Services. HCAHPS Summary Analyses [Internet]. Available at: https://hcahpsonline.org/en/summary-analyses/.

20. Reicheld FF. The One Number You Need to Grow [Internet]. Available from: https://hbr.org/2003/12/the-one-number-you-need-to-grow.

21. Centers for Medicare and Medicaid Services. HCAHPS Fact Sheet (CAHPS©Hospital Survey) October 2019 [Internet]. Available from: https://hcahpsonline.org/globalassets/hcahps/facts/hcahps_fact_sheet_october_2019.pdf.

22. Agency for Healthcare Research and Quality. Preparing Data from CAHPS®Surveys for Analysis [Internet]. Available from: https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/helpful-resources/analysis/preparing-data-for-analysis.pdf.

23. Centers for Medicare and Medicaid Services. HCAHPS Patient-Level Correlations [Internet]. Available from: https://hcahpsonline.org/globalassets/hcahps/summary-analyses/correlations/report_april_2018_corrs_pain_removed.pdf.

24. Kemp KA, Chan N, McCormack B, Santana MJ, Quan H. Correlation of inpatient experience survey items and domains with overall hospital rating. J Patient Exp, 2015;2(2). 10.1177/2374373515615977

    https://doi.org/10.1177/2374373515615977

25. Ahmed S, Kemp KA, Quan H, Johnson DW, Santana MJ. Identifying areas for improvement in pediatric inpatient care using the child HCAHPS survey. Paediatr Child Hlth, 2019;25(6):365–71. 10.1093/pch/pxz031

    https://doi.org/10.1093/pch/pxz031

26. Kemp K, Warren S, Chan N, McCormack B, Santana M, Quan H. Qualitative complaints and their relation to overall hospital rating using an H-CAHPS-derived instrument. BMJ Qual Safety, 2016;25:770–7. 10.1136/bmjqs-2015-004371

    https://doi.org/10.1136/bmjqs-2015-004371

27. Fairie P, Zhang Z, D’Souza AG, Walsh T, Quan H, Santana MJ. Categorising patient concerns using natural language processing techniques. BMJ Health Care Inform. 2021;28(1):e100274. 10.1136/bmjhci-2020-100274

    https://doi.org/10.1136/bmjhci-2020-100274

28. Ish D, Parker AM, Osoba OA, Elliott MN, Schlesinger M, Hays RD, et al. Using natural language processing to code patient experience narratives: capabilities and challenges [Internet]. Available from: https://www.rand.org/pubs/research_reports/RRA628-1.html.

29. Thiels CA, Hanson KT, Yost KJ, Zielinski MD, Habermann EB, Cima RB. Effect of hospital case mix on the Hospital Consumer Assessment of Healthcare Systems star scores: are all stars the same? Ann Surg. 2016;264(4):666–73. 10.1097/SLA.0000000000001847

    https://doi.org/10.1097/SLA.0000000000001847

30. Elliott MN, Zaslavsky AM, Goldstein E, Lehrman W, Hambarsoomians K, Beckett MK, et al. Effects of survey mode, patient mix, and nonresponse on CAHPS®hospital survey scores. Health Serv Res. 2009;44(2):501–18. 10.1111/j.1475-6773.2008.00914.x

    https://doi.org/10.1111/j.1475-6773.2008.00914.x

31. Toomey SL, Ellliott MN, Zaslavsky AM, Klein DJ, Ndon S, Hardy S, et al. Variation in family experience of pediatric inpatient care as measured by child HCAHPS. Pediatrics. 2017;139(4):e20163372. 10.1542/peds.2016-3372

    https://doi.org/10.1542/peds.2016-3372

32. Kemp KA, Chan N, McCormack B. The Alberta inpatient hospital experience survey: representativeness of sample and initial findings. Survey Prac, 2015;8(3). 10.29115/SP-2015-0012

    https://doi.org/10.29115/SP-2015-0012

33. Kemp KA, Chan N, McCormack B, Douglas-England K. Drivers of inpatient hospital experience using the HCAHPS survey in a Canadian setting. Health Serv Res 2015;50(4):982–97. 10.1111/1475-6773.12271

    https://doi.org/10.1111/1475-6773.12271

34. Steele BJ, Fairie P, Kemp KA, Santana MJ. Drivers of pediatric inpatient experience: retrospective analysis of case-mix factors for the Alberta pediatric inpatient experience survey in Alberta, Canada. BMJ Open. 2022;12(5):e048207. 10.1136/bmjopen-2020-048207

    https://doi.org/10.1136/bmjopen-2020-048207

35. Kemp KA, Ahmed S, Quan H, Johnson DW, Santana MJ. Family experiences of pediatric inpatient care in Alberta, Canada: results from the child HCAHPS. Hosp Peds, 2018;8(6):338–44. 10.1542/hpeds.2017-0191

    https://doi.org/10.1542/hpeds.2017-0191

36. Kemp KA, Fairie P, Steele BJ, Santana MJ. Adult experiences with hospitalization in Alberta, Canada during the COVID-19 pandemic: a comparative cross-sectional study. J Pt Exp. 2022;9:1–7. 10.1177/23743735221077518

    https://doi.org/10.1177/23743735221077518

37. Smiechowski J, Mercia M, Kemp KA, Oddone Paolucci E, Santana MJ, Kachra R. Using design-thinking to investigate and improve patient experience. PXJ. 2021;8(3):24–44. 10.1177/23743735221077518

    https://doi.org/10.1177/23743735221077518

38. Kemp KA, Santana MJ, Knudtson M, Oddone Paolucci E, Quan H. Patient-reported experiences with coronary artery bypass grafting and valve replacement. Can J Cardiol. 2019;35(10):1344–52. 10.1016/j.cjca.2019.05.008

    https://doi.org/10.1016/j.cjca.2019.05.008

39. Kemp KA, Fairie P, Steele BJ, Norris CM, Santana MJ. Sex differences in the experiences of patients hospitalized due to ischaemic heart disease in Alberta, Canada. CJC Open. 2021;3(12 Suppl):S36–S43. 10.1016/j.cjco.2021.08.011

    https://doi.org/10.1016/j.cjco.2021.08.011

40. Kemp KA, Fairie P, Steele B, McNeil D, Kromm S, Johnson DJ, Santana MJ. The experience of parents of hospitalized children living with medical complexity. Hosp Peds. 2021;11(10):1065–72. 10.1542/hpeds.2020-003038

    https://doi.org/10.1542/hpeds.2020-003038

41. Kemp KA, Southern D, McCormack B, Santana M, Quan H. Association of inpatient hospital experience with patient safety indicators: a cross-sectional, Canadian study. BMJ Open, 2016;6(7):e011242. 10.1136/bmjopen-2016-011242

    https://doi.org/10.1136/bmjopen-2016-011242

42. Kemp KA, Quan H, Santana MJ. Lack of patient involvement in care decisions and not receiving written discharge instructions are associated with unplanned readmissions up to one year. PXJ, 2017;4(2):13–22. 10.35680/2372-0247.1205

    https://doi.org/10.35680/2372-0247.1205

43. Kemp KA, Quan H, Fairie P, Santana MJ. Patient reports of night noise in hospitals are associated with unplanned readmissions among older adults. J Patient Exp. 2020;7(6):1425–31. 10.1177/2374373520916030

    https://doi.org/10.1177/2374373520916030

44. McAlister FA, Mu L, Bakal J, Kemp KA, Quan H. The Care Transitions Measure-3 is only weakly associated with post-discharge outcomes: a retrospective cohort study in 48,384 Albertans. J Gen Intern Med, 2019;34(11):2497–504. 10.1007/s11606-019-05260-8

    https://doi.org/10.1007/s11606-019-05260-8

45. Steele BJ, Kemp KA, Fairie P, Santana MJ. Proxy-rated pediatric health status is associated with unplanned health services utilization. Hosp Peds. 2022;12(1):61–70. 10.1542/hpeds.2020-005728

    https://doi.org/10.1542/hpeds.2020-005728

46. Sidhu VK, Bresee L, Kemp K, Koshman S, Pereira T, Neilson S. Accuracy of inpatient recall of interaction with a pharmacist: a validation study from two acute care teaching hospitals. J Pt Exp, 2019;6(1):62–7. 10.1177/2374373518771774

    https://doi.org/10.1177/2374373518771774

47. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:e001570. 10.1136/bmjopen-2012-001570

    https://doi.org/10.1136/bmjopen-2012-001570

48. Gleeson H, Calderon A, Swami V, Deighton J, Wolpert M, Edbrooke-Childs J. Systematic review of approaches to using patient experience data for quality improvement in healthcare settings. BMJ Open. 2016;6:e011907. 10.1136/bmjopen-2016-011907

    https://doi.org/10.1136/bmjopen-2016-011907

49. National Institute for Health Research. Improving Care by Using Patient Feedback [Internet]. Available from: https://evidence.nihr.ac.uk/wp-content/uploads/2020/03/Patient-Feedback-WEB-1.pdf.

50. Robert G, Cornwell J. Rethinking policy approaches to measuring and improving the patient experience. J Health Serv Res Policy. 2013;18(2):67–9. 10.1177/1355819612473583

    https://doi.org/10.1177/1355819612473583

51. Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a Science of Learning Systems: A Research Agenda for the High Functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43–50. 10.1136/amiajnl-2014-002977

    https://doi.org/10.1136/amiajnl-2014-002977