How to integrate diverse evidence: EBP+
How should we evaluate whether a policy works? Orthodox evidence-based approaches take Randomised Controlled Trials (RCTs) as the gold standard of evidence. But it is often impossible to conduct a high-quality trial that randomises individuals to a group that receives a policy intervention or a control group that receives an indistinguishable ‘placebo’ intervention. In practice we often need to look beyond RCTs and consider a much more diverse range of evidence. Some argue that we need to consider theory-based methods to understand the complexities of policy interventions. But it is unclear how to integrate these considerations into an overall judgement about whether a policy works.
EBP+ offers a broad framework within which both orthodox and theory-based methods can sit, and a systematic account of how to integrate and evaluate these different types of evidence in a consistent, unified and objective approach that increases public trust in evidence-based policy.
Evidential Pluralism as a methodology for integrating evidence
EBP+ is motivated by Evidential Pluralism, a philosophical theory of causal enquiry that has been developed over the last 15 years. Evidential Pluralism encompasses two key claims. The first, object pluralism, says that establishing that A is a cause of B (e.g., that a policy intervention causes a specific outcome) requires establishing both that A and B are appropriately correlated and that there is some mechanism which links the two that can account for the extent of the correlation. The second claim, study pluralism, maintains that evaluating whether A is a cause of B requires scrutinising both association studies (experimental and observational studies that repeatedly measure A and B, together with potential confounders, to measure their association) and mechanistic studies (studies of features of the mechanisms linking A to B).
Evidential Pluralism has previously been applied to medicine, where it leads to a development of evidence-based medicine called EBM+. The WHO International Agency for Research on Cancer uses this sort of approach to evaluate whether an environmental or occupational exposure causes cancer.
Evidential Pluralism has begun to be applied to evidence-based policy evaluation, leading to the EBP+ approach. EBP+ provides a systematic framework for integrating mechanistic evidence into policy evaluation.
Applying EBP+ in practice: Face masks
A narrow focus on experimental studies, especially RCTs, resulted in controversy and uncertainty concerning the effectiveness of public health interventions to reduce the spread of Covid-19, including public face mask mandates. This prompted calls for a more inclusive approach to evidence in responding to the novel, complex and rapidly changing problem of Covid-19 (Aronson et al., 2020; Greenhalgh et al., 2022; Mormina, 2022).
The causal claim of interest here is that a legal requirement to wear a face mask in public reduces the prevalence of symptomatic Covid-19 infections and thereby reduces the number of hospitalizations and deaths. The correlation claim is that a legal requirement to wear a face mask in public is negatively correlated with symptomatic infections, conditional on potential confounders.
A plausible mechanism hypothesis is that a legal requirement to wear a face mask in public increases the use of face masks which in turn reduces the prevalence of Covid-19 which reduces the prevalence of symptomatic covid infections. A hypothesised counteracting mechanism is that a legal requirement to wear a face mask in public will decrease compliance with other public health interventions, such as social distancing.
On the basis of available evidence, Trofimov and Williamson (forthcoming) find that quantitative studies detect a robust correlation across contexts. They also argue that a combination of quantitative and qualitative studies establishes the hypothesised mechanism of action, but undermines the hypothesised counteracting mechanism. Overall, they conclude that the combination of evidence of correlation and evidence of mechanisms establishes the effectiveness of face mask mandates.
Evidence gaps
EBP+ can also be used to point to gaps in the evidence base for a policy, and our introductory guide to EBP+ includes the additional examples of: online fake news and its effect on vaccination behaviour; minimum unit pricing for alcohol for reducing deaths; universal basic income; and awarding gaps in higher education.
