How do you think policy-makers and others weigh impact evaluation results?

Aidan Coville and I asked researchers to forecast findings from a discrete choice experiment through the Social Science Prediction Platform. A thread. 1/

First, for context, the original study asked policy-makers and practitioners at World Bank and Inter-American Development Bank workshops which of two programs they would prefer to implement. Each program had an impact evaluation attached to it, with varying attributes. 2/

For example, you could select a program done in your country with observational data and an estimated impact of improving school enrollment by 5 percentage points, +/- 1, or an RCT done in another region with some other estimated impact. 3/

We also asked how they would weigh this evidence from impact evaluations against advice from a local expert - since chances are that in any given situation you'll have local advice, but maybe only less relevant impact evaluation results. 4/

A few researchers took the survey at the workshops, but not many, so to help us compare weights across groups we asked researchers to go through the discrete choice questions as part of the later forecasting exercise and asked policy-makers and practitioners to do this, too. 5/

What did we find? As you might expect, the policy-makers put relatively more weight on factors associated with external validity (where it was done, whether it was recommended by a local expert), while researchers put more weight on internal validity (precision, method). 6/

BUT researchers completing the forecasting exercise also thought the researchers & policy-makers would behave more differently, in a sense: if you look at the rank-order of the attributes, they thought researchers would put the least (7th-most) weight on local advice... 7/

... when reality they put the 3rd-most weight on that factor, after the estimated impact and the result not being from an observational study. 8/

You can see full results in this updated paper: http://evavivalt.com/wp-content/uploads/How-Do-Policymakers-Update.pdf 9/9