Tolerance is not an absolute rule, but a social contract. Members of a tolerant society agree to tolerate others so long as others do the same. When someone violates the contract by being intolerant they cannot then proceed to hide behind that same contract for protection.

At some point a judgement has to be made about what is tolerant and what is not, and that is a judgement we make collectively as upholders of the social contract.