Beyond Baseline and Follow-up : The Case for More T in Experiments
The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.
Saved in:
| id |
dig-okr-109863403 |
|---|---|
| record_format |
koha |
| spelling |
dig-okr-1098634032024-08-08T15:44:26Z Beyond Baseline and Follow-up : The Case for More T in Experiments McKenzie, David AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations. 2012-03-19T18:01:51Z 2012-03-19T18:01:51Z 2011-04-01 http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20110425104143 https://hdl.handle.net/10986/3403 English Policy Research working paper ; no. WPS 5639 CC BY 3.0 IGO http://creativecommons.org/licenses/by/3.0/igo/ World Bank application/pdf text/plain |
| institution |
Banco Mundial |
| collection |
DSpace |
| country |
Estados Unidos |
| countrycode |
US |
| component |
Bibliográfico |
| access |
En linea |
| databasecode |
dig-okr |
| tag |
biblioteca |
| region |
America del Norte |
| libraryname |
Biblioteca del Banco Mundial |
| language |
English |
| topic |
AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH |
| spellingShingle |
AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH McKenzie, David Beyond Baseline and Follow-up : The Case for More T in Experiments |
| description |
The vast majority of randomized
experiments in economics rely on a single baseline and
single follow-up survey. If multiple follow-ups are
conducted, the reason is typically to examine the trajectory
of impact effects, so that in effect only one follow-up
round is being used to estimate each treatment effect of
interest. While such a design is suitable for study of
highly autocorrelated and relatively precisely measured
outcomes in the health and education domains, this paper
makes the case that it is unlikely to be optimal for
measuring noisy and relatively less autocorrelated outcomes
such as business profits, household incomes and
expenditures, and episodic health outcomes. Taking multiple
measurements of such outcomes at relatively short intervals
allows the researcher to average out noise, increasing
power. When the outcomes have low autocorrelation, it can
make sense to do no baseline at all. Moreover, the author
shows how for such outcomes, more power can be achieved with
multiple follow-ups than allocating the same total sample
size over a single follow-up and baseline. The analysis
highlights the large gains in power from ANCOVA rather than
difference-in-differences when autocorrelations are low and
a baseline is taken. The paper discusses the issues involved
in multiple measurements, and makes recommendations for the
design of experiments and related non-experimental impact evaluations. |
| topic_facet |
AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH |
| author |
McKenzie, David |
| author_facet |
McKenzie, David |
| author_sort |
McKenzie, David |
| title |
Beyond Baseline and Follow-up : The Case for More T in Experiments |
| title_short |
Beyond Baseline and Follow-up : The Case for More T in Experiments |
| title_full |
Beyond Baseline and Follow-up : The Case for More T in Experiments |
| title_fullStr |
Beyond Baseline and Follow-up : The Case for More T in Experiments |
| title_full_unstemmed |
Beyond Baseline and Follow-up : The Case for More T in Experiments |
| title_sort |
beyond baseline and follow-up : the case for more t in experiments |
| publishDate |
2011-04-01 |
| url |
http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20110425104143 https://hdl.handle.net/10986/3403 |
| work_keys_str_mv |
AT mckenziedavid beyondbaselineandfollowupthecaseformoretinexperiments |
| _version_ |
1807158876757295104 |