In Pursuit of Balance : Randomization in Practice in Development Field Experiments
Randomized experiments are increasingly used in development economics, with researchers now facing the question of not just whether to randomize, but how to do so. Pure random assignment guarantees that the treatment and control groups will have identical characteristics on average, but in any particular random allocation, the two groups will differ along some dimensions. Methods used to pursue greater balance include stratification, pair-wise matching, and re-randomization. This paper presents new evidence on the randomization methods used in existing randomized experiments, and carries out simulations in order to provide guidance for researchers. Three main results emerge. First, many researchers are not controlling for the method of randomization in their analysis. The authors show this leads to tests with incorrect size, and can result in lower power than if a pure random draw was used. Second, they find that in samples of 300 or more, the different randomization methods perform similarly in terms of achieving balance on many future outcomes of interest. However, for very persistent outcome variables and in smaller sample sizes, pair-wise matching and stratification perform best. Third, the analysis suggests that on balance the re-randomization methods common in practice are less desirable than other methods, such as matching.
Saved in:
| id |
dig-okr-109866910 |
|---|---|
| record_format |
koha |
| spelling |
dig-okr-1098669102024-08-08T16:34:40Z In Pursuit of Balance : Randomization in Practice in Development Field Experiments Bruhn, Miriam McKenzie, David Bruhn, Miriam ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY Randomized experiments are increasingly used in development economics, with researchers now facing the question of not just whether to randomize, but how to do so. Pure random assignment guarantees that the treatment and control groups will have identical characteristics on average, but in any particular random allocation, the two groups will differ along some dimensions. Methods used to pursue greater balance include stratification, pair-wise matching, and re-randomization. This paper presents new evidence on the randomization methods used in existing randomized experiments, and carries out simulations in order to provide guidance for researchers. Three main results emerge. First, many researchers are not controlling for the method of randomization in their analysis. The authors show this leads to tests with incorrect size, and can result in lower power than if a pure random draw was used. Second, they find that in samples of 300 or more, the different randomization methods perform similarly in terms of achieving balance on many future outcomes of interest. However, for very persistent outcome variables and in smaller sample sizes, pair-wise matching and stratification perform best. Third, the analysis suggests that on balance the re-randomization methods common in practice are less desirable than other methods, such as matching. 2012-06-01T19:07:20Z 2012-06-01T19:07:20Z 2008-10 http://documents.worldbank.org/curated/en/2008/10/9938284/pursuit-balance-randomization-practice-development-field-experiments https://hdl.handle.net/10986/6910 English Policy Research Working Paper; No. 4752 CC BY 3.0 IGO http://creativecommons.org/licenses/by/3.0/igo/ World Bank application/pdf text/plain World Bank, Washington, DC |
| institution |
Banco Mundial |
| collection |
DSpace |
| country |
Estados Unidos |
| countrycode |
US |
| component |
Bibliográfico |
| access |
En linea |
| databasecode |
dig-okr |
| tag |
biblioteca |
| region |
America del Norte |
| libraryname |
Biblioteca del Banco Mundial |
| language |
English |
| topic |
ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY |
| spellingShingle |
ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY Bruhn, Miriam McKenzie, David Bruhn, Miriam In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| description |
Randomized experiments are increasingly
used in development economics, with researchers now facing
the question of not just whether to randomize, but how to do
so. Pure random assignment guarantees that the treatment and
control groups will have identical characteristics on
average, but in any particular random allocation, the two
groups will differ along some dimensions. Methods used to
pursue greater balance include stratification, pair-wise
matching, and re-randomization. This paper presents new
evidence on the randomization methods used in existing
randomized experiments, and carries out simulations in order
to provide guidance for researchers. Three main results
emerge. First, many researchers are not controlling for the
method of randomization in their analysis. The authors show
this leads to tests with incorrect size, and can result in
lower power than if a pure random draw was used. Second,
they find that in samples of 300 or more, the different
randomization methods perform similarly in terms of
achieving balance on many future outcomes of interest.
However, for very persistent outcome variables and in
smaller sample sizes, pair-wise matching and stratification
perform best. Third, the analysis suggests that on balance
the re-randomization methods common in practice are less
desirable than other methods, such as matching. |
| topic_facet |
ALTERNATIVE APPROACH BOOTSTRAP DEVELOPMENT ECONOMICS DEVELOPMENT RESEARCH ECONOMIC MODELS ECONOMIC THEORY ECONOMICS LITERATURE ESTIMATORS EXPERIMENTAL DESIGN EXTERNALITIES FIELD EXPERIMENTS INCOME LINEAR MODELS LINEAR REGRESSION LOTTERIES METHODOLOGY MINIMIZATION MISSING OBSERVATIONS NORMAL DISTRIBUTIONS POLITICAL ECONOMY PRECISION PROBABILITY PROGRAMS RANDOMIZATION RESEARCH PAPERS RESEARCH REPORT RESEARCH WORKING PAPERS RESEARCHERS RISK AVERSION SAMPLE SIZE SCENARIO SCENARIOS SCIENTISTS SIGNIFICANCE LEVEL SIMULATION SIMULATIONS STANDARD DEVIATIONS STANDARD ERRORS STATA STATA CODE STATISTICAL ANALYSIS STATISTICAL METHODS STATISTICAL SIGNIFICANCE STATISTICAL TESTS TECHNIQUES VALIDITY |
| author |
Bruhn, Miriam McKenzie, David Bruhn, Miriam |
| author_facet |
Bruhn, Miriam McKenzie, David Bruhn, Miriam |
| author_sort |
Bruhn, Miriam |
| title |
In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| title_short |
In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| title_full |
In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| title_fullStr |
In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| title_full_unstemmed |
In Pursuit of Balance : Randomization in Practice in Development Field Experiments |
| title_sort |
in pursuit of balance : randomization in practice in development field experiments |
| publisher |
World Bank, Washington, DC |
| publishDate |
2008-10 |
| url |
http://documents.worldbank.org/curated/en/2008/10/9938284/pursuit-balance-randomization-practice-development-field-experiments https://hdl.handle.net/10986/6910 |
| work_keys_str_mv |
AT bruhnmiriam inpursuitofbalancerandomizationinpracticeindevelopmentfieldexperiments AT mckenziedavid inpursuitofbalancerandomizationinpracticeindevelopmentfieldexperiments AT bruhnmiriam inpursuitofbalancerandomizationinpracticeindevelopmentfieldexperiments |
| _version_ |
1807157151417761792 |