I completely understand. I'm hesistant to speak up too. I never know how they're going to react. Maybe, on the metro, people would come to my defense. Maybe.
Men protect each other, and out of fear, many of us don't say anything to protect ourselves or others.