What's wrong?

 A series of questions from a new survey sponsored by the Wall Street Journal is getting a lot of attention.  The figure that's been circulating shows the percent who says that the following values are very important "to you personally" in surveys taken in 1998, 2019, and 2023:

So it appears that patriotism, religion, having children, and community involvement are becoming less important to people, while money is becoming more important.  The WSJ story on the survey also mentions that the percent who say that tolerance and hard work is very important to them fell between 2019 and 2023.  So the results seem to show a dramatic turn towards selfishness in the past few years.

A few people have raised questions about this interpretation.  Patrick Ruffini notes that the 1998 and 2019 surveys were taken by telephone, while the 2023 survey was online.  He suggests that phone surveys are affected by "social desirability bias"--you want the interviewer to think well of you, so you give answers that will make you look better.  Social desirability bias might have inflated the numbers for values like hard work, patriotism, and community involvement, and held down the numbers for money.

Another new feature of the 2023 poll was that half of the sample were asked the response options ("very important," "somewhat important," "not that important," and "not at all important") in reverse order (see p. 7 here).  This may also make a difference--making a distinction "very important" and "somewhat important" may be harder than making a distinction between important and not important.  In this case, whether people say "very" or "somewhat" may depend on which answer is offered first.  In the phone surveys, "very" was always offered before "somewhat"--in half of the internet surveys, "somewhat" came first.  So my hypothesis is that the distribution of answers in 2023 differed depending on the order of the response options.  If so, comparison with the earlier years should be based only on the part of the sample that offers them in the same order as the phone surveys.  

Or you could look at the the combined percentages for "very" and "somewhat" important:

                                        1998    2019   2023
Hard work                         98%    98%   94%
Self-fulfillment                  94%   94%   91%
Tolerance                           NA     96%   90%
Money                                 88%   92%   90%

Community involvement   94%   96%    80%
Patriotism                           93%   88%    73%
Children                              84%    77%   65%
Belief in God                      NA     72%    65%
Religion                              89%    75%   60%

For the first four, there's little change--they are almost universally regarded as important at all three times.  The results for "self-fulfillment," which isn't mentioned in the WSJ article, are particularly interesting--the percent rating it as very important fell from 64% in 2019 to 53% in 2023.  That's hard to square with either the growing selfishness or the social desirability interpretations, but is consistent with my hypothesis.  These figures indicate some changes in the last few years, but not the general collapse of values that is being claimed.

Unfortunately, the Wall Street Journal doesn't make the original survey data available.  If they ever do, I will check and see if the response order makes a difference.