Can We Trust Shoestring Evaluations?
Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring," method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.
Summary: | Many more impact evaluations could be
done, and at lower unit cost, if evaluators could avoid the
need for baseline data using objective socio-economic
surveys and rely instead on retrospective subjective
questions on how outcomes have changed, asked
post-intervention. But would the results be reliable? This
paper tests a rapid-appraisal, "shoestring,"
method using subjective recall for welfare changes. The
recall data were collected at the end of a full-scale
evaluation of a large poor-area development program in
China. Qualitative recalls of how living standards have
changed are found to provide only weak and biased signals of
the changes in consumption as measured from contemporaneous
surveys. Importantly, the shoestring method was unable to
correct for the selective placement of the program favoring
poor villages. The results of this case study are not
encouraging for future applications of the shoestring
method, although similar tests are needed in other settings. |
---|