DEPARTMENT OF ECONOMICS AND FINANCE COLLEGE OF BUSINESS AND ECONOMICS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND
Replications in Economics: A Progress Report Maren Duvendack Richard W. Palmer-Jones W. Robert Reed
WORKING PAPER No. 26/2014
Department of Economics and Finance College of Business and Economics University of Canterbury Private Bag 4800, Christchurch New Zealand
WORKING PAPER No. 26/2014 Replications in Economics: A Progress Report Maren Duvendack Richard W. Palmer-Jones W. Robert Reed
December 3, 2014
Abstract: This study reports on various aspects of replication research in economics. It includes (i) a brief history of data sharing and replication; (ii) the results of the authors’ survey administered to the editors of all 333 “Economics” journals listed in Web of Science in December 2013; (iii) an analysis of 155 replication studies that have been published in peer-reviewed economics journals from 1977-2014; (iv) a discussion of the future of replication research in economics, and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena.
Keywords: Replication, data sharing, publication bias JEL Classifications: A1, B4
Acknowledgements: The authors wish to thank Richard Anderson and Bruce McCullough for comments on an earlier draft of this paper. We also wish to thank numerous journal editors who worked with us to ensure that our reporting of their replication policies was accurate. Cathy Kang, Akmal Fazleen, Alfred Zhao, and Sonja Marzi provided excellent research assistance. Financial support from the College of Business & Law at the University of Canterbury is gratefully acknowledged. 1 2
School of International Development, University of East Anglia, United Kingdom. Department of Economics and Finance, University of Canterbury, New Zealand
* Corresponding author: W. Robert Reed, Email: [email protected]
I. INTRODUCTION This study provides a progress report on the use of replications in economics. At least since the seminal study by Dewald et al. (1986), there has been a recognition in the economics profession that many of the empirical results in economics are not reproducible; and/or not generalizable to alternative empirical specifications, econometric procedures, extensions of the data, and other modifications to the original study. A survey of the current literature reveals that addressing this state of affairs has not been an easy task. While there have been substantial improvements in the sharing of data and code, publication in peer-reviewed journals of studies that replicate previous research is still a rare event. The concern that a substantial portion of empirical research is not reproducible and/or generalizable is not restricted to economics and the social sciences. A recent issue of Science was devoted to Data Replication and Reproducibility in the so-called “hard sciences.” 1 The concern with replication in science has become sufficiently widespread that it has crossed over to popular media. The Economist 2, the New Yorker 3, the Atlantic 4, BBC Radio 5, and the Los Angeles Times 6 are just a few of the popular media outlets that have recently reported on widespread concern with reproducibility in scientific research. And while popular interest tends to focus on academic fraud, others have pointed out that the nature of statistical analysis as it is practiced in applied disciplines such as economics is inclined to produce a
Science, 2 December 2011.
http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needschange-itself-how-science-goes-wrong, accessed 10 September 2014. 3
http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off, accessed 10 September 2014.