Post-publication review

 In 2019, I wrote about an article in the New York Times "1619 Project," which drew on a paper published in the Proceedings of the National Academy of Sciences in 2016:   "Racial Bias in Pain Assessment and Treatment Recommendations, and False Beliefs about Biological Differences between Blacks and Whites." The paper was based on a survey of medical students and residents that gave them hypothetical cases and asked them to rate how much pain they thought the patient would be feeling and what treatment they would recommend for the pain (narcotics vs. something weaker).*  According to the Times "when asked to imagine how much pain white or black patients experienced in hypothetical situations, the medical students and residents insisted that black people felt less pain."  Actually, the ratings were almost identical--the mean for black cases was 7.622 on a scale of 0-10, and the mean for whites was 7.626--so the description in the Times story was completely wrong.  Basically what the study found was that the number of false beliefs was associated with racial bias, but at the average level of false believes there was no bias in either direction.

 Last year I saw another Times story that referred to the paper as "an often-cited study," and I checked and found it had about 1400 citations, according to Google Scholar.  After the Supreme Court decision on affirmative action, I ran across another article that mentioned it (I forget where that appeared), which led me to check the citation count again.  It was up to almost 2000, and is now over 2,000.   That's a lot:  it ranks 6th out of the couple of thousand papers published in PNAS in 2016.  Presumably the 1619 project story helped to bring attention to it, but it was already doing well before then:  it had 25 citations in 2016, then 54, 103, and 152 in 2017-9.  

I was interested in seeing how well the academic literature did in describing the findings of the paper.   Google Scholar lists citing articles roughly in order of the citations that they have, so I started from the top and picked the first 20 with over 100 citations (a couple of books were listed, but I limited myself to journal articles).  

One of the citations could be called incidental:  “Contemporary, ‘mainstream’ epidemiology’s technocratic focus on individual-level biological and behavioral risk factors”

Four of them were accurate, in my judgment:  “a recent study showed that half of medical students and residents in their sample held biased beliefs such as ‘Black people’s skin is thicker than White people’s skin,’ assessed Black mock patients’ pain as lower than White mock patients, and subsequently made less accurate treatment recommendations for Black compared to White mock patients.”

“medical students who endorsed the false beliefs that Black patients had longer nerve endings and thicker skin than White patients also rated Black patients as feeling less pain and offered less accurate treatment recommendations in mock medical cases.”

 “in a 2016 study to assess racial attitudes, half of White medical students and residents held unfounded beliefs about intrinsic biologic differences between Black people and White people. These false beliefs were associated with assessments of Black patients’ pain as being less severe than that of White patients and with less appropriate treatment decisions for Black patients.”

“and a substantial number of medical students and trainees hold false beliefs about racial differences.”

Four were partly accurate:   “Implicit bias among clinicians and other healthcare workers can . . . contribute to . . . lower quality of care received . . . .”

“document false beliefs among medical students and residents regarding race-based biological differences in pain tolerance that resulted in racial differences in treatment.”

 “minorities . . . are less likely to have their pain appropriately diagnosed and effectively treated due to structural constraints, racialized stereotypes, and false beliefs regarding genetic differences on the part of health care providers.”

 “contemporary examples of anti-Black racism in healthcare in North America include racial bias in pain assessment and treatment recommendations between White and Black patients based on false beliefs about biological differences"

These correctly say that the study found evidence that views about biological differences were associated with differences in pain assessment and treatment, but fall short because they either imply that the study involved treatment of real cases or that it found differences in average levels of pain assessment.  

Ten were inaccurate--I don't mean that the statements are necessarily false, but the study provided no support for them:

“Racist beliefs among some providers that African Americans have unusually higher tolerance for pain . . .  may also have reduced opioid prescribing to African Americans relative to whites.”

 “false assumptions about Black-White physiologic differences in pain tolerance”

 “Black patients have been subjected to racially stratified diagnoses resulting in the denial of pain medication, based on the belief that they withstand pain better than other demarcated groups.” 

“Currently, Black patients also are less likely to receive . . . adequate doses of pain and cancer medication”

 “A substantial literature in psychology has documented physicians’ differential perceptions of Black patients in terms of . . . pain tolerance.”

 “there are known racial and socioeconomic biases in how a patient’s pain is perceived by observers”

 “Black patients are less likely to receive . . . accurate diagnoses (e.g., pain assessments)”

 “Demographic factors associated with chronic pain and its undertreatment include . . . being an African American or other underrepresented minority . . . .”

 “The backdrop for such discussions includes systemic and pervasive racial biases in the US healthcare system, including lack of insurance and a lesser quality of care for non-white, rural, and low-income populations”

 “perceptions of suffering are shaped by various factors such as the victim’s race”

The most common mistakes were citing it as evidence of racial differences in pain assessments or in prescription of pain medication (rather than recommendations for hypothetical cases).

And one cited this study while describing a completely different one:  “Black mothers in the wealthiest neighborhoods in Brooklyn, New York have worse outcomes than white, Hispanic, and Asian mothers in the poorest ones, …. likely due to societal bias that impacts Black women.”

I don't know what's typical, but more than 50% inaccurate citations is disturbing.   Another striking thing was that I didn't find any efforts to replicate the study.  It had an obvious limitation:  the sample was just students and residents at one medical school.  So it would be natural to try to replicate it at other medical schools, or among practicing physicians.  You could also go beyond straight replication, and do things like consider other hypothetical cases (e. g., ones that were more ambiguous), or the possibility of interactions between race and other factors like gender.  The data were from an online survey, so replicating it would be cheap and easy--I could see giving it as a project for a master's student or even an undergraduate.  Of course, I can't say that there are no published replications, but I made enough effort to be confident that there aren't many.  Is that because of a lack of attempts, or because attempts haven't found anything, so they haven't been published?**

*There was also a survey of Mechanical Turk participants, but that doesn't get much attention.  

**The evidence in the original study was weak--there's a good chance that it's just a combination of random variation and what Andrew Gelman calls "researcher degrees of freedom."