Pandemics causing death are linked with increases in support for radical political movements, all the way back to the 1918 flu pandemic and the birth of 20th century fascism
I wonder if it's a result of feelings of powerlessness and just a general impression that government is ineffective. It's almost on par with a foreign invasion, but it just seems like anything we do is just to try to mitigate or contain the effects. We can't actually "fight" these diseases as a society, it's this incredibly individual battle that each of us goes through, some of us make it, some of us don't, and that's it. You can wear a mask or reduce your exposure, or the government can give out tests, monetary aid, provide information updates, or whatever, but none of that tackles it head on, it's natural disaster that washes across the entire country. So people just get frustrated with it and realize that government can't really help them that much.
I think it is more about the fear of receiving harm from "others" outside their local group and placing the blame on those outsiders. Like blaming Gypsies/Travelers and other nomadic groups for things when they are around, or the antisemetic hatred based on myths of being society manipulated by Jews.
Pandemics remind people that there is a world out there that could affect them negatively and they want a group to be a part of and people outside that group to blame.
Obviously not universally true for all people, but a significant portion.