It’s important to strike the right balance in research assessment between stories and numbers: between expert assessment and having a broad range of data and indicators to spot gaps and challenge bias. We’ve undoubtedly missed this mark before: it seems likely that a more positive research culture should involve rather less of a focus on citation metrics.
But when assessing policy impact I think we’re faced with the opposite problem: too many stories and not enough data.
To be fair impact is probably something that’s impossible to measure objectively, so not focusing on numbers is understandable. But sometimes it’s about the journey not the destination, and about making sure that you’ve packed enough clean socks.
Overton.io is a new project that tries to track the evidence and influence that goes into policy. It does that by systematically collecting and indexing large numbers of policy documents and the news stories, think tank reports, working papers & scholarly outputs surrounding them.
It launched earlier this year as a web application that’s free for academic research purposes but supported by paid subscriptions from universities and think tanks (Overton has no external funding, so it’s supported entirely by its users).
In the simplest case it’s able to show you all of the papers from a particular university that were cited by documents on GOV.UK, or which academics were mentioned in Hansard or in select committees. Often, though, it’s the network of links between different items that’s more useful: being able to trace the path of academic research picked up by a think tank policy brief that then informs a government white paper.
The aim isn’t to produce some new all singing all dancing metric to replace expert judgement – that’d be a terrible mistake – but to give you tools and data that help you:
* See when university/policy interactions are happening, as they happen
*… and where they should be happening, but aren’t
* Quickly find or supplement evidence for existing case studies
* Gauge if the volume of interactions in a given topic are higher or lower than you might expect
Figuring out if and how interactions actually lead to useful impact remains difficult, but at least we can take some of the legwork and researcher reporting burden out of finding them: a worthy goal.
Overton doesn’t do anything that you couldn’t with Google and enough time, but it does do it on a much larger scale – it’s far, far bigger than other policy indexes, covering 180+ countries and thousands of individual sources.
That said collecting and curating the data has been fraught with difficulties, not least because different contexts require different definitions of what a policy document actually is. It’s hard to collect and index policy from some countries, for various reasons: from developing nations like Togo, but also from much larger states like China. Your research could be very popular amongst policymakers there and Overton might never know.
Even in the UK and US there are still and probably always will be gaps in the data: policy related documents or workshop reports kept private, informal advice on what not to do, all the interactions that are never recorded.
That means we’ve got to be careful of unintended consequences, especially early on while we’re all still figuring out the framework for interpreting the data. One important next step is to figure out how to signpost caveats and highlight best practice clearly for users, and we’ve been leaning heavily on early adopters to help with this.
On that note we’re keen to get feedback from UPEN members; just drop me a note on euan@overton.io if you’d like access!
Euan Adie runs Overton.io. He previously founded Altmetric, the alternative metrics provider, and worked at Digital Science and Nature Publishing Group.
