Thursday, June 12, 2014

The Story of the Hurricane

I was looking at Andrew Gelman's blog yesterday and saw a post on a study saying that hurricanes with female names caused more deaths than hurricanes with male names.  The study came out a couple of weeks ago; I seem to recall hearing some news reports and assuming there was probably something wrong with it, but I didn't give it any more thought.  This morning I looked at the New York Times and saw that Nicholas Kristof gave about half of his column to uncritically recounting the claims of the study. Then I looked a little more and saw that there had been a lot of news coverage, and a lot of critical commentary.  But the criticisms seemed sort of peripheral, or raised questions without really identifying a specific flaw.  So I read the original paper, downloaded the data, and did some analysis.  Their claim does not stand up, and here is my attempt to explain why not.  It's possible that someone beat me to it (in fact, I hope someone did, since the problem was so basic), but given the nature of the internet the more places it appears the better.

A statistical model uses variables to predict the value of another variable (total deaths resulting from the hurricane).  The "deviance" is a measure of how much of total deaths is not predicted.  So the goal is to get a small deviance using a small number of predictors.  


Here are the deviance and number of predictors in two of their models:

Deviance         Predictors
136.1                 3          female name, storm damage,                                        barometric pressure
121.8                 5          ""  plus interactions (products)                                    of female and damage, female                                    and pressure


Here are the deviance and number of predictors from two alternative models that I fit:

Deviance         Predictors
   97.5                 3        female name, logarithm of storm                                  damage, barometric pressure
   95.3                 5        ""  plus interactions of female                                  and log damage, female and                                        pressure

The models using the logarithm rather than the original variable had much lower deviance.  Adding the two interactions to the model with the logarithm reduced the deviance by 2.2, but the usual standard is that adding two predictors has to reduce the deviance by at least 6 to qualify as evidence that there's anything there (ie a reduction of less than 6 is not "statistically significant").  So the best model has a deviance of 97.5 and three predictors.  In that model, the estimated effect of the "femaleness" of the name (which they treat as a matter of degree) is .024, with a standard error of .036, which is not statistically significant, or close to statistically significant.  

So the flaw was that they controlled for the dollar value of damage when they should have controlled for the logarithm of damage.    With the right control, there is no evidence that the gender of the name makes any difference.  

Notes:  1. the paper and data, published in the Proceedings of the National Academy of Sciences



2 comments:

  1. As Krugman (sorry, Kristof; sometimes they are difficult to distinguish in their obtuseness) writes, the takeaway from the study was supposed to be that female-named hurricanes caused more damage than male-named hurricanes because our patriarchal and sexist society doesn't take women as seriously as men and so underestimates the likely impact of a hurricane with a female name. You're generous in having assumed that there was something wrong with the study; when I read the initial flurry of news reports about it my reaction was that the theory was so insane that it was not even wrong.

    Since you have the data downloaded and ready for analysis, I'm curious. How does, say, the day of the week on which the hurricane made landfall (or, better yet, the date of the month, since everyone in hurricane country can be reliably counted on to be in church on Sunday) come out as a predictor compared to the femaleness of the name?

    ReplyDelete
  2. The data set doesn't include that information, so fortunately I can't spend time checking that. However, given the nature of their model, there's a good chance that the interaction of anything with storm damage would have a "significant" effect.


    As far as the general idea of the study, there is a lot of experimental evidence that factors that should be irrelevant if people were rational actually can affect perceptions. Then it's reasonable to suspect that the irrelevant factors might make a difference in the real world. The reason I was skeptical is that there are so many influences on hurricane deaths that if there's any effect it should be small--maybe a fraction of a percent. Any effect that would show up in 92 cases would have to be really big.

    ReplyDelete