From bias adjustment to bias avoidance: an unexpected journey

Music City Center 

The suite of 80+ ONS business surveys have developed organically over the past 80+ years, with different launch dates meaning their statistical methods were optimized for a variety of data collection processes and to evolving guidelines for best practice. This has led to a lack of consistency, and esoteric practices, which presents challenges both in ongoing support and transformation activities, such as combining existing surveys to meet new requirements.

A specific example of the problems this evolution caused was the Business Enterprise Research and Development survey (BERD). Comparisons with a similar output, produced by a different government department, showed an increasing divergence over time. The root cause was found to be an incomplete sampling frame developed specifically for the BERD survey, which had worked historically due to targeting the specific businesses engaged in R&D activity, but had failed to adapt when the economic landscape changed and many more small businesses started undertaking R&D. Complex bias adjustment methodology was developed, which accounted for multiple specific methods issues unique to the survey, including non-response in a source responsible for updating the incomplete frame.

Following lengthy consultation, communication with interested bodies both nationally and internationally, and thorough internal QA and scrutiny, the bias adjustment was successfully implemented and nine years of data were revised and republished in 2022. The new figures attracted immense public and press interest, which although diminished over time, still makes headlines regularly. After some early surprise despite our extensive pre-release efforts to inform our users, the new figures have been widely welcomed and seen as an improvement.

To continue the improvement, the BERD survey was radically redesigned for 2023, again with extensive QA. The approach taken was to apply a robust sample design, which would be easily combined with other standard ONS surveys, and implement modern best practice methods throughout. One of the biggest changes was to increase the sample size ten-fold, and sample directly from the ONS business register to identify R&D hotspots. The sample increase, and associated costs, were planned for the first year only – after which sample optimisation based on the first year's data aimed to reduce the sample by half. How this succeeded will be revealed in this talk.

The approach taken for BERD blazed the trail for a wider holistic transformation of ONS business surveys, ongoing since late 2023, with the aims of increased quality, reduced business burden, improved response rates, consistent and best practice methods. The vision is of integrated modular surveys built on a core foundation of our largest annual survey, updated and re-invigorated for that over-arching purpose.

Bias

Frame

Methodology

Sampling

Revision

Future