Developing an Evaluation Toolkit for Data Linkage Pipelines

Abstract

The linkage evaluation toolkit project aims to develop tools and criteria that can be used to evaluate data linkage pipelines and methods. This toolkit will not only allow assessment of a single pipeline, but will also facilitate the comparison between pipelines; enabling analysts to identify their strengths and weaknesses. There are currently limited standardised evaluation metrics for linkage pipelines, except for the precision, recall, and bias of the linked datasets. Eight key evaluation areas were identified; accuracy, bias in linkage error, flexibility in input datasets, scalability, efficiency, platform suitability, ease of use, and the ease and transparency of quality assurance. Evaluation criteria for each area were identified and quantified. Tools are being developed to enable users to assess pipelines across these criteria. These include tools for stress-testing, testing the pipelines under different parameters, assessing bias , and capturing and quantifying qualitative feedback. We are liaising with quality teams for standardised tools to assess precision and recall. Once finalised, the toolkit will be used to compare linkage pipelines when linking the same administrative datasets, to identify which pipeline performs best across each criterion. Future use will include identifying suitable linkage methods to support the UK’s 2031 Census. The evaluation toolkit will allow analysts to evaluate the strengths and weaknesses of linkage pipelines, in both their use and impact on linked outputs. The methods can be used to improve existing pipelines, and to support the development and evaluation of new methodologies in future.