Can We Trust Shoestring Evaluations?

Can We Trust Shoestring Evaluations?

Show other versions (1)

Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring," method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.

Saved in:
Bibliographic Details
Main Author: Ravallion, Martin
Language:English
Published: 2012-03-01
Subjects:AGRICULTURE, ANTIPOVERTY PROGRAM, CLINICS, CORRELATIONS, DATA COLLECTION, DEBT, DEVELOPMENT POLICY, DEVELOPMENT PROJECTS, DEVELOPMENT RESEARCH, DWELLING, ECONOMIC RESEARCH, ECONOMIC SURVEYS, ESTIMATORS, EXTERNALITIES, GENDER, GROWTH MODELS, HOUSEHOLD SURVEYS, HOUSEHOLDS, HOUSING, HYGIENE, IMPACT EVALUATION, INCOME, INTEREST RATE, INTERVENTION, INTERVENTIONS, INTERVIEWS, LABOR MOBILITY, LEARNING, LIVING CONDITIONS, LIVING STANDARDS, LONGITUDINAL DATA, MARGINAL COST, OCCUPATION, POVERTY REDUCTION, PRODUCTION COSTS, PRODUCTION INPUTS, PROGRAM EVALUATION, PROGRAMS, QUANTITATIVE DATA, QUESTIONNAIRES, RELIABILITY, RESEARCH METHODS, RESEARCH WORKING PAPERS, SAFETY, SELECTION BIAS, SOCIAL SCIENCE, SURVEY DATA, SURVEY DESIGN, TARGETING, TREATMENT EFFECTS, URBAN AREAS, VILLAGES, WEIGHTING, WELFARE LEVELS,
Online Access:http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20120305094659
https://hdl.handle.net/10986/3269
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring," method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.

Similar Items