IJPDS International Journal of Population Data Science

https://doi.org/10.23889/ijpds.v8i1.2118.review.r1.reviewb

Thank you for the opportunity to review this interesting piece of research. Overall I found the concept of the study a valid and useful one, however there were parts where the methods became lost in translation and difficult to follow. I feel that the paper would benefit from an overhaul to ensure that the text is clear and methods transparent.

I will present this review based upon the sections within the reviewers’ guidelines.

Problem Statement, Conceptual Framework, and Research Question

In essence it was clear what the purpose of the study was. The introduction provided a clear background and a case for the research question. I was also able to understand the question posed.

Reference to the Literature and Documentation

The references drawn upon seem both sensible, in context and reasonably up to date. They are correctly referred to within the text and it is clear how this supports the research question.

Relevance

The study seems relevant to the journal and its audience and addresses an important issue within many research studies – that of identifying the correct prevalence and population. It adds to the literature as it considers outcomes not previously studied. There are some generalisability issues which are discussed in the discussion

Research Design

The research design is good, however there are places where the text needs clarification as it is hard to follow. The design is appropriate for the question. Biases within the methods are intrinsic to the study question and as such are referred to within the discussion.

Instrumentation, Data Collection, and Quality Control

The methods of data collection are adequately described, however again in places they are hard to follow.

Population and Sample

There are places within the methods where results have got mixed in, for example unless the inclusion/exclusion criteria is those aged 0 to 112 this is a result, and if it is part of the criteria then why 112? I feel that this text could be simplified which would make it easier to follow.

Data Analysis and Statistics

The analysis section should be relabelled to indicate that this is part of the methods. It needs to be made clear that the gold standard is a composite of data from APCDC, EDDC, RBDM, CODURF, i.e. the occurrence of the outcome of interest in any of these datasets. The formulae does not add anything to the explanation, a suitable reference could be used for those wishing to follow the methods in that depth. More information needs to be provided around the confidence intervals, where was it applicable to adjust them for clustering by practice site. I’ve looked at a few of the confidence intervals and haven’t been able to replicate any -this may well be down to lack of clarity and explanation rather than incorrect values. I also think that the study would benefit from more descriptive statistics within the text.

Reporting of Statistical Analyses

• Please give a measure of spread in the text for age.
• How many general practices are included?
• Please quote the national estimates – do the figures for this population sit in the confidence intervals around the estimates?
• The details of the selection process should be presented before you talk about the comparative numbers of outcomes in each dataset.
• When there is only male and female there is no need to quote both.
• Can you describe the times in a continuous measure as well as being categorised?

Presentation of Results

Generally the results can be followed, however there are places where the text is hard to follow and I feel more clarity is need. I have also indicated above places where more information would provide context to the results.

Discussion and Conclusion: Interpretation

The conclusions are clearly stated, however it is unclear where they are leading. They summarise the study well and how they support information already known. However it just slightly leaves me feeling, what next? The implications for current research are demonstrated, but are there any suggestions or direction for improvement?

Title, Authors, and Abstract

The title is clear and informative, whilst being representative of the study. However it doesn’t indicate that what the study is doing is seeing how well the outcomes are matched across two datasets. The authorship is appropriate. The abstract provides a concise and accurate summary of the main paper.

Presentation and Documentation

As mentioned previously there are some parts in the text where potential errors have caused loss of clarity, for example

“However, limited such linkage has occurred to date in Australia.”

Do you mean: “However limited, such linkage has occurred to date in Australia?” This sentence isn’t clear.

“The decision to link datasets therefore requires a good understanding of the value add of bringing them together to answer specific questions.“

Some restructure within the methods section could make the paper easier to follow and the methods more replicable.

https://doi.org/10.23889/ijpds.v8i1.2118.review.r1.reviewc

This is an important topic with the growing interest in the use of electronic medicla records (or EHR as this paper alludes to them) from primary care for research and health services planning. This paper has several limitations (addressed below) but is an addition to the literature. The paper is well written, includes references to the literature and provides analyses of the comparisons of the data extraxcted from a single EMR in Australia (a limitation) and adminstrative data availbale for the same population (another limitation as recognized by the authors). Two previous studies that validate chronic diseases in the same EMR are referenced but none of the other similar research done with different healthcare systems and EMRs.

It is important to recognize that the purpose for the use of data is an important factor in choosing the most approriate data source. For some questions the sensitivity of the data is more important while for others the specificity may serve the purpose more appropriately. This study has identified poor sensitivity for the chosen major health issues when compared to administrative data, but high specificity and related metrics. Ultimately the paper recognizes the need to supplement EMR extracts for some analyses. This is however not always or even feasible requiring a more nuanced recognition from the authors.

Specific comments:

Patients: It was clear to me how data from patients who may have attended more than clinic using the same EMR were handeled.

Outcome variables: There is reference to "Pyefinch (used in Best Practice) and Docle (used in Medical Director) codes". Please provide more context and references to these.

Analysis: The table (Table 1) introducing the metrics used does not belong in this paper. These are widely used definitions. The timing of the event in relation to the results is relevant and may be of interest but is not well justified. There is no justification for the inclusion of that one factor specifically.

Results: Tabl2 includes 95% CI for each of the results. This is not the correct way to report descriptive analyses. confidence intervals apply to estimates based on analyses. These are not estimates. For some means and SD may apply as was provided for age.

Discussion: The first sentence is problematic. Firstly this study does not describe health outcomes, but acute events. The outcomes associated with these events are not addressed. Secondly there are two papers referenced where chronic disease algorithms (10) and death - atrue outcome (20) using data from this EMR are validated. The discussion identifies limitations (such as the timing of the illness in relation to the study) which could have been addressed through a sensitivity analysis. How would the inclusion of those cases in the EMR data have changed the outcome metrics? This would not have required a huge amount of extra work.

The strengths of this study suggests that this is the first to include the data used as the valicdation data. This statement is strange in the light of two facts. First, the authors in=dentify the reference standards as being limited as not true "gold standards". Second they do reference two previous algorithm validation studies. Perhaps this should be explained in the discussion.

Conclusions: Once again the acute events studied in this paper are referred to as outcomes. The conclusion section also introduces the potential value of including EMRs in research into health outcomes (in combination with administrative data) due to the ability to control for confounding factors. This conclusion is not based on this study. This concept may be acceptable in the discussion but should not be claimed as a conclusion for this study.

Thank you for your email of 22 November 2022 regarding our manuscript, “Identification of acute serious outcomes using linked Australian general practice, hospital, emergency department and death data: implications for research and surveillance”. We are pleased to resubmit our edited manuscript for publication in the International Journal of Population Data Science. We have addressed the reviewer’s additional concerns as outlined below.

Reviewer 1

1. There are places within the methods where results have got mixed in, for example unless the inclusion/exclusion criteria is those aged 0 to 112 this is a result, and if it is part of the criteria then why 112? I feel that this text could be simplified which would make it easier to follow.

Routinely collected data includes errors (eg. year of birth 200 years ago or in the future), therefore we must apply acceptable data limits. MedicineInsight records year (but not date) of birth, and age is considered valid where it ranges between 0 and 112 years (inclusive) as calculated at 1st July 2020. 112 is included as the upper limit based on the oldest recorded person currently living in Australia being 111 years. We have amended the Patients subsection under Study population and study period to read:

The study cohort included regular patients with valid age (0 to 112 years) with at least 3 clinical encounters between 1 January 2019 and 31 December 2020 at an eligible MedicineInsight practice in NSW which met the data quality requirements in February 2022 [5] and with at least one hospital, ED or mortality record between 1 January 2010 and 31 December 2020.

2. The analysis section should be relabelled to indicate that this is part of the methods.

The analysis subheading has been removed, incorporating that section with methods.

3. It needs to be made clear that the gold standard is a composite of data from APCDC, EDDC, RBDM, CODURF, i.e. the occurrence of the outcome of interest in any of these datasets.

The assessment of agreement regarding fact of event has been amended to: For each acute serious event we calculated percentage of agreement, sensitivity, specificity, negative and positive predictive values (NPV/PPV)[10, 26] of the MedicineInsight algorithms compared with an external reference standard during 2019 and 2020. The external reference standard was a composite of APDC, EDDC, RBDM and CODURF. An outcome recorded in any of these datasets was considered to have truly occurred.

4. The formulae does not add anything to the explanation, a suitable reference could be used for those wishing to follow the methods in that depth.

Table 1 (calculations for measures of agreement) has been removed and replaced with references [10] and [27].

5. More information needs to be provided around the confidence intervals, where was it applicable to adjust them for clustering by practice site. I’ve looked at a few of the confidence intervals and haven’t been able to replicate any -this may well be down to lack of clarity and explanation rather than incorrect values.

In the last paragraph of the methods section the description of confidence intervals has been expanded to read as follows:

As the data are clustered within practices, variance was adjusted to account for correlation between observations within clusters, and confidence intervals adjusted accordingly, using SURVEYFREQ and SURVEYMEANS procedures in SAS.

6. I also think that the study would benefit from more descriptive statistics within the text.

We have attempted to bolster descriptive statistics presented within the text in the results section, particularly within the first paragraph of that section. We have aimed to limit repetition of statistics presented in tables for readability and to adhere to journal word limit suggestions.

7. Please give a measure of spread in the text for age.

Median age, quartile 1 and quartile 3 (in years) have been included in Table 2

8. How many general practices are included?

61, this has been added to the text in the first paragraph of the results section.

9. Please quote the national estimates – do the figures for this population sit in the confidence intervals around the estimates?

National estimates of people visiting an Australian general practice at least once during 2019-20 are available from Medicare Benefits Schedule (MBS) data, and have been added to table 2 and in text within the first paragraph of the results section.

10. The details of the selection process should be presented before you talk about the comparative numbers of outcomes in each dataset.

Details of the selection process are described in the methods and in Figure 1 of the results. In the interest of succinctness and readability we would prefer not to repeat the same information in the results text that is already available in Figure 1.

11. When there is only male and female there is no need to quote both.

We have removed Males from table 1 and quote only females in the results section, paragraph 1.

12. Can you describe the times in a continuous measure as well as being categorised?

The median and interquartile (Q1 and Q3) limits of the difference between the MedicineInsight and reference index dates (per event, rounded to whole days) have been added to table 4 and the methods and results have been updated to reflect this.

13. The title is clear and informative, whilst being representative of the study. However it doesn’t indicate that what the study is doing is seeing how well the outcomes are matched across two datasets.

Title changed to “ Agreement of acute serious events recorded across datasets using linked Australian general practice, hospital, emergency department and death data: implications for research and surveillance.”

14. As mentioned previously there are some parts in the text where potential errors have caused loss of clarity, for example

“However, limited such linkage has occurred to date in Australia.”

Do you mean: “However limited, such linkage has occurred to date in Australia?” This sentence isn’t clear.

“The decision to link datasets therefore requires a good understanding of the value add of bringing them together to answer specific questions. “

We have attempted to clarify the passage the reviewer indicated above by rewording as follows:

Linking general practice data with other routinely collected data, such as hospital admissions, emergency department (ED) and death records, improves their utility for research, monitoring and surveillance. Despite these potential benefits, to date, very few linkages of this kind have been performed in Australia. Technical and governance issues, while not insurmountable, take time and considerable resources to overcome [9]. A good understanding of the contribution of different datasets is important prior to linking them.

15. Discussion and conclusions: The implications for current research are demonstrated, but are there any suggestions or direction for improvement?

Our findings suggest that general practice records will complement hospital, ED and mortality data, and that both are required for robust ascertainment of acute serious events in any future studies. We have noted this in the discussion and conclusions and have made some changes to these sections to try to improve clarity as per both reviewers’ feedback.

Reviewer 2

1. The paper is well written, includes references to the literature and provides analyses of the comparisons of the data extracted from a single EMR in Australia (a limitation) and administrative data available for the same population (another limitation as recognized by the authors).

In the ‘Data Sources’ section about MedicineInsight, we have added a reference to Best Practice and Medical Director being the two Clinical Information software systems that contribute to MedicineInsight, reading:

It uses third-party data extraction tools [17, 18] which de-identify, extract and securely transmit data from the Best Practice (BP)™ or Medical Director (MD)™clinical information systems, for harmonisation, cleaning and storage [5].

2. Two previous studies that validate chronic diseases in the same EMR are referenced but none of the other similar research done with different healthcare systems and EMRs.

We have referenced similar research from different healthcare systems in the third paragraph of the introduction, as follows:

Primary care data from other countries has reasonable recording of acute events [13-16]. However, health care systems differ, and this question hasn’t been examined specifically for Australia.

3. Outcome variables: There is reference to "Pyefinch (used in Best Practice) and Docle (used in Medical Director) codes". Please provide more context and references to these.

In the ‘Data Sources’ section about MedicineInsight, we have added a reference to Best Practice and Medical Director being the two Clinical Information software systems that contribute to MedicineInsight. We have also added the following statement about Docle and Pyefinch codes to the Outcome definitions subsection, with a reference provided:

Docle and Pyefinch are Australian general practice coding systems which consist of clinical terminologies for diseases, clinical findings and therapies [26].

4. Analysis: The table (Table 1) introducing the metrics used does not belong in this paper. These are widely used definitions.

Table 1 (calculations for measures of agreement) has been removed and replaced with references [10] and [27].

5. The timing of the event in relation to the results is relevant and may be of interest but is not well justified. There is no justification for the inclusion of that one factor specifically.

We have added a statement to the last paragraph of the Introduction reading as follows: We also assessed timing of events which is important for some time-sensitive research questions.

6. Results: Tabl2 includes 95% CI for each of the results. This is not the correct way to report descriptive analyses. confidence intervals apply to estimates based on analyses. These are not estimates. For some means and SD may apply as was provided for age.

We have removed the 95% Cis in Table 2 . Median, quartile 1 and quartile 3 age in years has been added. National estimates of people visiting an Australian general practice at least once during 2019-20 are available in Medicare Benefits Schedule (MBS) data, and have been added to table 2 and in text within the first paragraph of the results section.

7. Discussion: The first sentence is problematic. Firstly this study does not describe health outcomes, but acute events. The outcomes associated with these events are not addressed.

The manuscript has been amended throughout to refer to ‘acute serious events’.

8. Secondly there are two papers referenced where chronic disease algorithms (10) and death - a true outcome (20) using data from this EMR are validated. The discussion identifies limitations (such as the timing of the illness in relation to the study) which could have been addressed through a sensitivity analysis. How would the inclusion of those cases in the EMR data have changed the outcome metrics? This would not have required a huge amount of extra work.

We agree with the reviewer’s commentary that a sensitivity analysis may have produced improved results addressing limitations such as the timing of illness in relation to the study. Given the majority of events were recorded within a 30 day period, we expect that the lack of data from 2018 and 2021 should impact a small proportion of events during January 2019 and December 2020. We do not therefore expect that a sensitivity analysis including data from 2018 and 2021 would change the conclusions, as this only amends a small proportion of events occurring for two months out of the 24 month study period.

9. The strengths of this study suggests that this is the first to include the data used as the validation data. This statement is strange in the light of two facts. First, the authors identify the reference standards as being limited as not true "gold standards". Second they do reference two previous algorithm validation studies. Perhaps this should be explained in the discussion.

We have expanded the first paragraph of the strengths and limitations subsection of the discussion to read as follows:

This is the second Australian study [20], to our knowledge, to use full-scale record linkage to compare acute serious events between general practice EHRs and population-based data collections. Coded diagnoses from NSW hospital admissions, emergency department and cause of death data was used as the reference standard against which accuracy of the diagnostic algorithms for the acute serious conditions was benchmarked. The majority of validation studies of primary care EHR use reviews of EHR as the reference standard [29, 34]. However, as EHR reviews are time consuming and labour intensive, the sample size is often small which limits the generalisability of results [10]. Linking to external data sources as the reference standard enables a much larger sample size and improves study power and representativeness.

10. Conclusions: Once again the acute events studied in this paper are referred to as outcomes. The conclusion section also introduces the potential value of including EMRs in research into health outcomes (in combination with administrative data) due to the ability to control for confounding factors. This conclusion is not based on this study. This concept may be acceptable in the discussion but should not be claimed as a conclusion for this study.

We have moved this point to the third paragraph of the discussion, reading:

General practice EHRs contain detailed patient information on potential confounding factors unavailable in other administrative datasets. Using linked general practice, hospital, ED and mortality data is therefore recommended in research on acute serious events to accurately ascertain cases, build appropriate patient cohorts and more adequately control for confounding. Our findings suggest that general practice records will complement hospital, ED and mortality data, and that both are required for robust ascertainment of acute serious events. Overall, our findings support the use of general practice EHRs for research, monitoring or surveillance of the selected acute events, only when linked to hospital, ED and mortality data.

My coauthors and I look forward to your decision. Please do not hesitate to contact me should you require any further information or edits.