A statistical model uses variables to predict the value of another variable (total deaths resulting from the hurricane). The "deviance" is a measure of how much of total deaths is not predicted. So the goal is to get a small deviance using a small number of predictors.
Here are the deviance and number of predictors in two of their models:
Deviance Predictors
136.1 3 female name, storm damage, barometric pressure
121.8 5 "" plus interactions (products) of female and damage, female and pressure
Here are the deviance and number of predictors from two alternative models that I fit:
Deviance Predictors
97.5 3 female name, logarithm of storm damage, barometric pressure
95.3 5 "" plus interactions of female and log damage, female and pressure
The models using the logarithm rather than the original variable had much lower deviance. Adding the two interactions to the model with the logarithm reduced the deviance by 2.2, but the usual standard is that adding two predictors has to reduce the deviance by at least 6 to qualify as evidence that there's anything there (ie a reduction of less than 6 is not "statistically significant"). So the best model has a deviance of 97.5 and three predictors. In that model, the estimated effect of the "femaleness" of the name (which they treat as a matter of degree) is .024, with a standard error of .036, which is not statistically significant, or close to statistically significant.
So the flaw was that they controlled for the dollar value of damage when they should have controlled for the logarithm of damage. With the right control, there is no evidence that the gender of the name makes any difference.
Notes: 1. the paper and data, published in the Proceedings of the National Academy of Sciences
2. Jeremy Freese has a number of interesting comments on the study.
As Krugman (sorry, Kristof; sometimes they are difficult to distinguish in their obtuseness) writes, the takeaway from the study was supposed to be that female-named hurricanes caused more damage than male-named hurricanes because our patriarchal and sexist society doesn't take women as seriously as men and so underestimates the likely impact of a hurricane with a female name. You're generous in having assumed that there was something wrong with the study; when I read the initial flurry of news reports about it my reaction was that the theory was so insane that it was not even wrong.
ReplyDeleteSince you have the data downloaded and ready for analysis, I'm curious. How does, say, the day of the week on which the hurricane made landfall (or, better yet, the date of the month, since everyone in hurricane country can be reliably counted on to be in church on Sunday) come out as a predictor compared to the femaleness of the name?
The data set doesn't include that information, so fortunately I can't spend time checking that. However, given the nature of their model, there's a good chance that the interaction of anything with storm damage would have a "significant" effect.
ReplyDeleteAs far as the general idea of the study, there is a lot of experimental evidence that factors that should be irrelevant if people were rational actually can affect perceptions. Then it's reasonable to suspect that the irrelevant factors might make a difference in the real world. The reason I was skeptical is that there are so many influences on hurricane deaths that if there's any effect it should be small--maybe a fraction of a percent. Any effect that would show up in 92 cases would have to be really big.