30 November 2017

Evidence-map to close gap between UNICEF's activities and the evidence behind them

This news is redirected by The Secretariat for Development Cooperation at SCIENCE from SciDevNet.

UNICEF has been testing an evidence-mapping tool to bridge a “big disconnect” between its activities and the evidence behind them, according to Kerry Albright, chief of Research Facilitation and Knowledge Management at the UN agency’s Office of Research-Innocenti.

“Across large parts of our organisation, [the] instructional incentives, time and space to slow down, take stock, learn lessons and draw upon the available global evidence base for improved decision-making are still evolving,” she told SciDev.Net.

Albright made the remarks as part of a consultation on an evidence mapping tool that she said is part of a broader push to build a stronger evidence culture at the agency. She cautioned that her comments were her personal views and did not represent the views of UNICEF.

The tool, called Evidence Gap Map (EGM), is a matrix designed to help policymakers and practitioners quickly see where high-quality studies are plentiful, or where more research is needed. It indicates the availability of evidence — not what it says — from various sources, not just peer-reviewed studies.

“The global evidence base of effective interventions to meet the … challenges facing children is mostly weak, scattered and too poorly translated to be usable,” Albright told SciDev.Net. “Many widely-used approaches are unsupported by rigorous evidence.”

"Many widely-used approaches are unsupported by rigorous evidence

Kerry Albright

At UNICEF, this has partly to do with the agency’s original service delivery mandate that goes back to the aftermath of the World War II, according to Albright.

Working with the Campbell Collaboration — a social sciences organisation which produces the mapping tools with 3ie, the International Initiative for Impact Evaluation — is a small contribution to the larger evidence puzzle, she explained.

Barriers, duplication The overall goal is to generate the practical knowledge needed to better understand what works, and to understand barriers to implementation of interventions. This ultimately avoids the costs of duplication and enables the agency to reach more children, said Albright.

But to get there, there are hurdles to overcome. Agency staff currently have trouble even getting their hands on evidence they need – it’s only available through a research access account managed “on a shoestring”, Albright explained.

Laurence Chandy, UNICEF’s director of data, research and policy, added that staff are so burdened with operational work that they simply do not have the time to read about the latest studies. This means the evidence used in programmes can be outdated.

The UN agency is one of the organisations assessing how useful the mapping tool can be in practice. They also include the International Rescue Committee (IRC), Sightsavers, the Beam Exchange and USAID.

At a consultation this month (10 November), which was part of 3ie’s London Evidence Week, Annalisa Brusatti, Senior Technical Advisor for Child Protection at IRC, echoed the need for evidence to improve the effectiveness of interventions on the ground. But in practice, there are limitations, she said.

Brusatti, who shared early experiences from the IRC’s use of the tool, said that although the maps make finding evidence easier, they often fail to capture the context that makes evidence meaningful for staff who need to use it on the ground.

Yet Howard White, CEO of the Campbell Collaboration, says that although context matters and can affect the impact of an intervention, its role can be overstated or misunderstood.

“[The] IRC transplanted a developed-country model of parenting programmes to developing countries,” he noted. But although parents in some cultures may not believe in stimulating infants with things like storytelling and play, the evidence shows that psychosocial development interventions do help cognitive development, he explained.

“So it is important to understand what elements of a programme are transferable and in what ways they need to be adapted,” said White.

Both the IRC and UNICEF continue to promote use of the tool internally. “There are still relatively few evidence gaps maps related to international development,” says Albright.

UNICEF is developing a mega-map of child welfare interventions that is planned for completion next year. This is a type of map that shows gaps in evidence synthesis by factoring in systematic reviews and gap maps, but no primary studies.

Development of the mapping tools is a work in process. According to White, feedback from users suggests they prefer maps that are easy to understand and enable them to “drill down to the underlying study”.