The (Lack of) Impact of Impact: Why Impact Evaluations Seldom Lead to Evidence-based Policymaking

Abstract

A recurring puzzle to many academics and some policymakers is why impact evaluations, which have become something of a cottage industry in the development field, have so little impact on actual policymaking. In this paper, I study the impact of impact evaluations. I show, in a simple Bayesian framework embedded within a standard contest success function-based model of competition amongst anti-evaluation policymakers, Bayesian policymakers, and frequentist evaluators, that the likelihood of a program being cancelled is a decreasing function both of the impact estimated by the evaluation and of the prior on whose basis the program was approved to begin with. Moreover, the probability of cancellation is a decreasing function of the effectiveness of the influence exerted by frequentist evaluators. Since the latter?s effectiveness in terms of lobbying in favor of their findings in the real world is likely to be close to zero, the likelihood of cancelling a program that was approved in the first place, despite its suffering a highly negative evaluation, is extremely low. The model thus provides one possible explanation for why impact evaluations have so little impact in the realm of decisionmaking, and why they have contributed so little to evidence-based policymaking.