I find the mask thing difficult in terms of worry etc. I fully accept that there are some people that can't wear masks. I accept that that includes people that have psychological reasons for not wearing them and people that have been unable to get suitable masks for their needs, in addition to those who can't physically wear one at all.
At the same time, based on my previous experiences of postviral fatigue, and very aware of people's attitudes towards fatigue conditions, I'm utterly terrified of catching the virus. Further, when people don't wear masks, whilst I don't direct my feelings to particular strangers, nevertheless if I'm somewhere where it's realistic to suppose that a large proportion of those without masks are not genuinely entitled to be not wearing them, e.g. on a bus where over half the passengers, maybe even most of the passengers, aren't wearing a mask, I feel threatened by them, without knowing which ones genuinely can't wear a mask, which don't believe masks make any difference, and which ones have an attitude that "I don't care if I pass on this virus and you die."
I have had people, particularly men, come up so close to me in places like checkouts that I could see their breath on my glasses and I've responded by saying clearly and firmly "If I can't trust you not to get close enough to me not to pass on a deadly virus, I certainly don't trust you not to get close enough to me to grope me or knock me over."
And that's how I feel. The people getting close enough to me in the street or shops to bump into me and cause me to stumble don't wear masks.
Where I am, the majority of people I encounter, particularly in the residential side roads, are very considerate, as we play the socially-distancing dance in and out of driveways. But this is a densely populated urban area and I don't necessarily go just in my own neighbourhood.
By analogy, it's like the very small proportion of people that dash to help me if I fall over who are what I'll call 'rescuer muggers' and 'rescuer gropers'. Few and far between, but they exist. That makes falling over more scary than it should be. Ditto sitting near maskless people on the bus.