I wonder why they are afraid of difference in others

Either through experience and unfair generalization or being taught be someone else. What I mean, specifically, is... say it's pre-covid, and I see a guy with a mask running into a store. If the only times before that I saw guys with masks running into stores was on the 9 o'clock news and those guys were robbers, I might be inclined to use stereotyping rather than open ended logic. And it makes sense in survival situations. But now in 2024, we ll have a lot of experience seeing people with masks in normal ituations. This can be extended to any et of visual traits and any stereotypable situation, if people are hammered with enough repetition. So, back in the early 90's, many Americans only ever saw bronze skinned people on the news in regard to terrorism, so there was a prevalent stereotype to the point that many people immediately assumed the Oklahoma City bombing was middle eastern people. It ended up being rural white farmers who did it. Good people, bad people, etc., all come in different shapes, colors, and sizes, but not everyone encounters all sorts of people, so people form stereotypes based off of limited knowledge (i.e. ignorance). It's all survival instincts that served us well eons ago: don't go near the brightly striped snake, don't touch the spiky plant, etc., but, as we stopped gathering food and moved into cities, the skills required for survival changed a lot and we only changed a little.

I understand about the surviving you describe but everyone will encounter a social situation in which they will have a hostile or disrespectful reaction to someone different from them. It is ignorance on their part.