Synthetic controls: a new approach to evaluating interventions

This working paper explores the potential for applying synthetic control methods to place-based interventions within Scotland, making use of the increasing availability of routinely collected data.

Summary

Synthetic control methods are a novel approach to comparative case study research using observational data. Though developed within political science, the methods can potentially be applied to a wide range of evaluation problems in economics, public health, social policy and other disciplines.

In the traditional approach, an area in which a new or redesigned service is being implemented is compared with another ‘control’ area (in which there is no change) and statistical adjustment used to account for any differences between areas that might bias the comparison.

In the new approach, a synthetic control is derived using data on past trends in all potentially comparable areas, providing a more robust basis for identifying the impact of the service change.

Synthetic control methods may be a valuable addition to the range of techniques available for non-randomised evaluations of social, economic and public health interventions.

To date there have been few applications in a UK context, and none in Scotland. Published evidence suggests considerable potential to apply synthetic controls to public service innovations at NHS Board, local authority or Community Planning Partnership level, and may widen the range of policy and practice changes that can usefully be evaluated.

This working paper explores the potential for applying synthetic control methods to place-based interventions within Scotland, making use of the increasing availability of routinely collected data. If synthetic control methods can be usefully applied, we will have identified an efficient solution to a wide range of pressing evaluation problems.

Section 2 provides a brief summary of the synthetic control method. Section 3 describes the range of applications of the method to date. Section 4 looks in greater detail at some applications that are particularly relevant to evaluating public service reform. Section 5 concludes with some recommendations for the use of synthetic controls in Scotland.

The Appendix summarises how the literature search for this working paper was carried out.