Study highlights lack of transparency in real-world evidence research
Transparency about the methods used by scientists is fundamental to trust in research findings. This is particularly important at a time when some scientific articles have been based on fabricated or unethically collected data, when AI can be used to generate fake articles, when the number of scientific article retractions now exceeds 63,000, and when expert data scientists using an identical dataset and identical protocol can generate disparate results.
The consequences of non-transparent and non-reproducable research extend beyond academia. When research findings are used to justify testing new therapies, halt clinical trials, or inform changes to clinical practice or government policy, a lack of transparency can have significant real-world implications.
In a new study published in the International Journal of Population Data Science (IJPDS), researchers from the University of New South Wales, Sydney examined the uptake of open science practices in articles published in IJPDS between 2019 and 2024. The journal publishes international research using large-scale population and administrative data.
Journals and academic institutions are increasingly encouraging or requireing researchers to adopt open science practices. These include five key practices: providing a link to the study protocol, pre-registering the protocol, including a statement on data availability, sharing analytical code, and citing reporting checklists or guidelines used in preparing the article.
However, the study found that none of the 41 eligible articles followed all five practices, and there was no evidence of adoption of these practices over time.
Open science practices are designed to improve transparency, reproducibility, reliability and collaboration in research. According to the study’s lead author, Ms Lillian Liu, “the findings offer stark evidence of the gap between what the population data science profession should be doing and what it is currently doing”.
The authors argue that greater commitment to openly sharing research methods - including analytical code and, where legally permitted, underlying data - is needed. They also highlight the importance of enhanced training and education, stronger researcher networks where good practices can be shared, incentives for openness, and journal requirements that encourage or mandate transparency.
With the recent release of the Australian National Health and Medical Research Council (NHMRC) and Medical Research Future Fund (MRFF) Open Science policy - which broadens open science expectations to include code sharing in line with the FAIR4Rs principles - this study provides a timely reminder of how much work remains to be done.
Click here to read the full article
Ms Lillian Liu, Project Manager, Centre for Big Data Research in Health, UNSW Sydney