In a number of posts, I have argued that opposition to immigration is not primarily a cover for racial prejudice--it's an issue in its own right. There are substantial race/ethnic differences on immigration issues today. In August 2016, Pew asked "All in all, would you favor or oppose building a wall along the entire border with Mexico?" 43% of non-Hispanic whites were in favor, compared to only 19% of blacks and 22% of Hispanics. But in 2006, a Gallup/USA Today poll asked "Do you favor or oppose each of the following proposals as a way to reduce illegal immigration to the United States? How about... building a wall along the border with Mexico?" 40% of non-Hispanic whites were in favor, as were 53% of blacks and 40% of Hispanics. The differences as a whole were not statistically significant, although if you just consider the black/white difference it was borderline.
What happened in those 10 years? The most straightforward explanation is that building a wall went from an idea that didn't receive serious attention to the favorite policy of the Republican candidate for president, and something that all prominent Democrats condemned. Blacks and Hispanics are overwhelming Democrats, so they followed their party (or reacted against the party they disliked).
PS: I got the idea for this post when I read something that said Donald Trump was a "white nationalist" and mentioned the wall as one of the pieces of evidence for this. I couldn't remember exactly where, but I did a search for the phrase in the New York Times and got the following results:
[Data from the Roper Center for Public Opinion Research]