Mistakes were made

Going into the election, Hillary Clinton had a lead of 3.3% in the polls, according to the Real Clear Politics average.  In the popular vote, she had a lead of 0.2%, as of this writing.  Was this a "failure of polling," as a New York Times story says?  The Gallup poll gives a list of its last poll before each election since 1936 and the actual election results which I summarize below as the difference between Democratic lead in the two-party vote in the poll and the election.  A positive sign means that the Democrats did better in the election than in the last poll; a negative sign that they did worse.

Year   Error
2016 -3.1
2012 +5
2008 -4
2004 -2.4
2000 +2.5
1996 -3.1
1992 -6.4
1988  5.1
1984  0.4
1980 -6.8
1976 +3
1972 +0.4
1968 +0.4
1964 -5.4
1960 -0.8
1956  3.4
1952 -7.8
1948  9.4
1944  4.6
1940  6
1936 13.6

The error has two parts--one is from sampling, and the other is from everything else--for example, supporters of the different candidates might differ in their willingness to respond, or the formulas for calculating turnout might be be wrong in ways that favored one party over the other.  The size of sampling error can be calculated from statistical theory--the other part can't, and has to be estimated from experience.  

The combined sample for the RCP average is so large that sampling error is trivial, on the order of 0.1%.  The samples for the individual Gallup polls are smaller, so sampling error is a factor.  However, it's not nearly big enough to account for the discrepancies.  A rough calculation is that the non-sampling error has a standard deviation of about 5.  Until 1948, the Gallup poll used quota samples.  That period included two big errors--the notorious 1948 election, and the 1936 election, when the poll showed a Democratic lead of +12 (56%-44%), but the actual lead was +25 (62.5%-37.5%).  If you confine it to the 1952-2012 period, the non-sampling error has a standard deviation of about 3--that is, candidate leading by three in the polls would lose about one time in six if the errors are normally distributed.  Most of the prediction sites seemed to give Clinton about an 85% chance of winning, which was just about right according to this analysis. Someone just going by the polls would have been a little surprised by the outcome, but not shocked.

Ironically, the problem may have been that journalists (and other observers, including me) paid too much attention to other "information" beyond the polls.  Similarly, in the Brexit referendum, the perception of the result as a shock wasn't based on the polls, which showed a close contest, but on a general reading of the public mood.