I honestly think it’s because you’re physically reminding them that Covid still exists and it makes them deeply uncomfortable. My wife and I wear one for similar reasons and she’s been accosted a few times where as I haven’t. It’s disgusting behaviour.