Or, The Short Version of Why Open Access Matters
In Open Access, Accountability, and For-Profit publishing, I wrote about how the lack of model transparency and access to data prevented researchers from replicating others' results, leading to an over-reliance on the peer-review process to vouch for a piece of research's validity. I also mentioned how peer-reviewers are not required to audit the veracity of a model or verify its results, and that this lack of oversight could result in inaccurate information influencing policymakers.
Now, researchers Thomas Herndon, Michael Ash, and Robert Pollin have called into question the conclusions of the economics paper "Growth in a Time of Debt" for exactly these reasons. Written in 2010 by Carmen Reinhart (University of Maryland) and Kenneth Rogoff (Harvard), the paper has played a not-insignificant role in the public discourse on economic policy during the current recession. As Mike Konczal of Next New Deal summarizes:
This has been one of the most cited stats in the public debate during the Great Recession. Paul Ryan's Path to Prosperity budget states their study "found conclusive empirical evidence that [debt] exceeding 90 percent of the economy has a significant negative effect on economic growth." The Washington Post editorial board takes it as an economic consensus view, stating that "debt-to-GDP could keep rising — and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth."Their findings also provided the basis of Reinhart's testimony before the Senate Budget Committee on Feb. 9, 2010, and have been used to support the argument for austerity in Europe and the United States.
Initially it would seem that access to data isn't an issue, as Reinhart and Rogoff provide the historical data they used along with their sources on their website. But publicly-available data is meaningless without context, and the authors don't provide any clarity on which dataseries and methodology they used. This is why model access is equally important to experiment repeatability and accountability. (There is some discussion on whether or not "Growth in a Time of Debt" was ever actually peer-reviewed, but as noted above, there's a good chance it would have passed review without anyone ever having to look at the model and datasets anyway.)
From Herndon, Ash and Pollin (pg 5):
We were unable to replicate the RR results from the publicly available country spreadsheet data although our initial results from the publicly available data closely resemble the results we ultimately present as correct. Reinhart and Rogoff kindly provided us with the working spreadsheet from the RR analysis. With the working spreadsheet, we were able to approximate losely the published RR results. While using RR's working spreadsheet, we identifi ed coding errors, selective exclusion of available data, and unconventional weighting of summary statistics.Reinhart and Rogoff's work demonstrated a seemingly straight-forward, data-driven analysis linking causation from a country's public debt to GDP growth, which is great talking-head bait and support for austerity economics. But, as succinctly put by Herndon, Ash and Pollin, "A necessary condition for a stylized fact is accuracy".
Update: Now with video!
Reinhart/Rogoff debacle reaches fever pitch with Steven Colbert going to town.