Policymakers have ball-park mechanisms to figure out what is ‘going on out there’, but the sure-fire way to be certain whether a policy is achieving its intended outcomes and for whom, is to evaluate and learn lessons. So what are the challenges of assessing the impact of social, economic and public health policies – and how can researchers using longitudinal surveys help?

According to the National Audit Office, only 8% of major government projects are robustly evaluated, while 64% are not evaluated at all. So it is not always clear how key policies and programmes are making a difference to people’s lives. With the UK government spending something in the region of £1 trillion a year, it is important to know how well different policies and programmes are working. Learning lessons plays a vital role in the durability, coherence and legitimacy of policies in driving change.

But the perpetual drive for new ideas, and the challenges of measuring impact, mean that evaluations are under-used. Indeed, complex modern societal problems require continual framing, intervention, learning and experimentation - there are rarely complete solutions. Moreover, civil servants argue that the focus on delivery, ‘fire-fighting’, or bureaucracy also make it harder to focus on outcomes. So policies risk being transactional rather than transformative. The NAO investigation identified both demand side and supply side challenges in government, but there are also challenges of data and measurement.

In the field of social, economic and environmental development, policy proposals can rarely be tested – and those that can test often opt for randomised controlled trials. However, observational data also have an important role to play, including in measuring spill-over effects and unintended consequences. Here we are primarily discussing interventions that affect thousands or millions of lives, and it’s not as simple as whether something works or not. A range of factors and processes, such as social and cultural norms, class and economic systems, the way different stakeholders respond to new policies and how something is implemented locally, can affect impact. In some cases, results may not be apparent for years, and what we are looking for is a slowing down or acceleration of change. 

Some initiatives are building better evidence for decision making, such as the What Works Network, changes to HM Treasury’s Green Book and Magenta Book, guidance for civil servants, and the establishment of the Evaluation Task Force – and Understanding Society has started a project on how long-term panel data could help.

Evaluations are more likely to be feasible, and the results used, if they are designed into the policy from the start. According to the Department for Levelling-Up, Housing and Communities, “the advantage of considering evaluation evidence from the outset is that it increases the likelihood of generating timely and helpful information to assist in delivering the department’s objectives”.

Longitudinal surveys like Understanding Society, or repeated data, can make a valuable contribution to evaluations, because data is collected from the same set of individuals over time, and across the entire year, allowing comparisons before and after a (large-scale) intervention or between groups (where one group is not affected by a policy). The Study collects data from across the UK, enabling the evaluation of policies that are devolved, or gradually rolled out across the country.

How evaluation research can generate impact

Robust evaluations can be difficult, especially in a dynamic environment, with many ‘moving parts’ to people’s lives. This is precisely where researchers excel. For example, when doing research they understand problems with confounding factors when trying to understand cause and effect.

Different evaluation methods can be used with longitudinal data, offering varying degrees of robustness: from pre and post analysis and interrupted time series analysis, to difference in difference methods and regression discontinuity design, to name some common research methods. Longitudinal studies can also be a cost-effective option – providing data which might otherwise cost the evaluation process a lot to collect.

However, even with large panel surveys like Understanding Society, identifying an intervention group can be a constraint. It is usually not feasible to ask extensive questions to identify if respondents have been the beneficiary of a range of policies or services, and where the target of policy is quite narrow, it is unlikely that a population survey will provide enough sample numbers to assess impact.

Linking survey and administrative data can help with the ‘identification problem’, where information on take-up or use of a service is captured in administrative data sources. For example, evaluating the impact of free-schools meals on child development, health and educational outcomes is feasible as a result of linking Understanding Society data with the National Pupil Database (NPD) for England.

Understanding Society data has been used to assess the impact of a number of other government policies, too.

The ‘bedroom tax’

One of the advantages of longitudinal data is that researchers can assess impact by looking at people’s lives before and after a policy was implemented. The first of our policy evaluation case studies summarised how research used Understanding Society data from 2009 to 2015 to look at housing tenants who received housing benefit in order to look at the ‘under-occupancy penalty’ before and after its introduction in 2013.

What quickly became known as the ‘bedroom tax’ reduced housing benefits for social tenants deemed to have a ‘spare’ bedroom. It aimed to cut the costs of social housing, and to match households to properties better.

The research found that the housing benefit bill fell by around £350 million, but that this was only 70% of the projected savings – and the government was forced to spend an extra £60 million in discretionary payments to protect people from the impacts of the policy.

Those affected saw their income fall by around 3.5% – a figure the researchers described as “not negligible for such a low-income group”. The policy had some success, then, but it was distinctly qualified.

Same-sex marriage

The results of policies may also be affected by factors outside the scope of the legislation. The aim of the Marriage (Same Sex Couples) Act 2013 was to equalise the rights of gay and lesbian people – and we might expect that to have a natural effect on mental health for people whose lives are changed by it.

Research using Understanding Society showed that mental health improved for all sexual minorities when marriage equality was introduced, but the improvement was biggest for men who also reported good levels of family support.

This was another policy evaluation that benefitted from the longitudinal nature of the data, using Waves 3-7 of the Study’s main survey, covering 2011-17, before and after the legislation. The improvement in mental health was lowest among sexual minority men who had previously reported below-average levels of family support. The mental health of sexual minority women also improved, but family support was less of a factor.

Pensions and health equality

Well-intentioned policies can have positive or negative unintended consequences, sometimes described as ‘spill-over effects’. One example of an unintended good consequence was the introduction of automatic pension enrolment after the 2008 Pensions Act which has reduced a previously unmeasured mental health gap.

Before automatic enrolment, people with poor mental health – particularly men – were more likely to work for employers who were less likely to offer a workplace pension scheme. Men with poor mental health working in the private sector were 3.7% less likely to take part in a workplace pension scheme, while female private sector employees with poor mental health were 2.9% less likely to participate. After the legislation took effect in 2012, the mental health disparity in pension participation disappeared – and Understanding Society allowed the researchers to eliminate other factors which might have explained the gap.

Benefitting from Understanding Society

If you are interested in learning more about Understanding Society and how your research community can use the survey, please get in touch with our Policy & Partnerships Unit. The sheer range of questions we ask – on education, employment, health and wellbeing, income and deprivation, family life, and environmental attitudes and behaviours, among other subjects – means researchers can use the data to explore how policies touch on some very different areas of life.

Anyone interested in the Make Surveys Sexy Again event will also find these blogs from Understanding Society, the UK Household Longitudinal Study, interesting. Also on the topic of evaluation and impact measurement, Raj Patel and Chris Coates from the Study look at the challenges of policy evaluation, and how longitudinal surveys like Understanding Society can be used to assess impact – with examples of use cases. There is growing recognition that ‘doing government’ need to improve, with the need for long-term strategic work while dealing with shorter-term pressures. For example, the Commons Liaison Committee is currently undertaking an inquiry into the government’s approach to strategic thinking and strategy-making, and what over-sight role committees can play - with Professor Matthew Flanders, UPEN chair, providing evidence to the Committee.