Thursday, February 28, 2013

On the Errors of SuperFreakonomics


In a must-read article in the recent American Scientist, Andrew Gelman and Kaiser Fung summarizes the take-downs of several error-prone examples in SuperFreakonomics, the new 'pop science' book by Levitt and Dubner (authors of the popular Freakonomics book), and the associated Freakonomics blog.
An extended quote about a sample case:
Monica Das Gupta is a World Bank researcher [...] attributed the abnormally high ratio of boy-to-girl births in Asian countries to a preference for sons, which manifests in selective abortion and, possibly, infanticide. [...] Emily Oster (now a professor at the University of Chicago) attacked this conventional wisdom. [...] Dubner and Levitt praised Oster and her study, [...] published in the Journal of Political Economy during Levitt’s tenure as editor:
[Oster] measured the incidence of hepatitis B in the populations of China, India, Pakistan, Egypt, Bangladesh, and other countries where mothers gave birth to an unnaturally high number of boys. Sure enough, the regions with the most hepatitis B were the regions with the most “missing” women. Except the women weren’t really missing at all, for they had never been born.
Oster [...] eventually [she] admitted that the subject-matter experts had been right all along. One of Das Gupta’s many convincing counterpoints was a graph showing that in Taiwan, the ratio of boys to girls was near the natural rate for first and second babies (106:100) but not for third babies (112:100); this pattern held up with or without hepatitis B.
In a follow-up blog post, Levitt applauded Oster for bravery in admitting her mistake, but he never credited Das Gupta for her superior work. Our point is not that Das Gupta had to be right and Oster wrong, but that Levitt and Dubner, in their celebration of economics and economists, suspended their critical thinking.
The article is a reminder of the perils of 'popularizing science' for all of us who attempt dissemination of science in some form or other. A relevant passage from a separate section that discusses the perils of pop-science:
Success comes at a cost: The constraints of producing continuous content for a blog or website and meeting publisher’s deadlines may have adverse effects on accuracy. The strongest parts of the original Freakonomics book revolved around Levitt’s own peer-reviewed research. In contrast, the Freakonomics blog features the work of Levitt’s friends, and SuperFreakonomics relies heavily on anecdotes, gee-whiz technology reporting and work by Levitt’s friends and colleagues. Just like good science, good writing takes time.
I liked the 'blog' part of that quote.

3 Comments:

  1. gaddeswarup said...

    Arunn, this article was linked in a December 2011 post of Gelman. I wonder why it took so long to appear
    http://andrewgelman.com/2011/12/15/freakonomics-what-went-wrong/

  2. Arunn said...

    Swarup: Sorry for the 'recent' word in the post (struck off now). The article did appear in the AS Jan/Feb 2012 issue. I just wanted to link it now for a planned subsequent post.

    Regards,
    Arunn

  3. gaddeswarup said...

    Oops. I misread. Even after looking at the article, I thought it was a 2013 article. Some implicit expectation I guess. Happening often lately, signs of age.