If everyone is tolerant of every idea, then intolerant ideas will emerge. Tolerant people will tolerate this intolerance, and the intolerant people will not tolerate the tolerant people.
Someone else being a twat won't make me violate my principles. I'm not good to others because they're good to me. I'm good to others because they're an end themselves, not a means to my ends.
If you are good to nazi's because they are good to you, regardless of what they do to others, Then your principles, and you as a person, are shit, and you should be treated as nothing but an infiltrator for their cause, because that is what you are.
It's not nearly that childish. I was talking about treating non-violent people well and you jumped straight to Hitler. Can we acknowledge there's a pretty big difference there?
Despite being against the death penalty I still feel robust self defense is essential. If someone is attacking you, being good to them is applying the appropriate level of force and not going out of your way to harm them extra for funsies. Even in the rare situation I'm fine with people being deleted, it should be that mundane. No torture. No pain. No consideration for what they've done. It's a practical necessity.
And that's completely your right to do. However, that is not what the tolerance contract covers. It goes beyond what most people would tolerate normally. Also, people cannot both break the social contract, and then insist you hold up the other end.
By example, I've previously had long debates over nazi Germany and Hitler's economic recovery. I would even tolerate Nazis, if they followed the social contract from their side. Unfortunately, the various Nazis groups regularly break that contract. They then try and hide behind it, when others take offence.
Conversely, I also disagree with the "tankies". They tend not to break the social contract however. This gives them the right to reasonable tolerance of them, and their views. They respect others, despite disagreeing with them. They, in turn, gain a level of respect in discussions.
Don't get me wrong, I am tolerant of a lot, from purely moralistic reasoning. The social contract is a larger entity however. It formalises what many of us feel. It also shows us where the lines are, beyond which people are abusing our tolerance. It's the larger social version of our internal morals.
I don't find social contract arguments all that convincing, but we can just pretend my social contract is "no violence or you get fucked" and ignore that. Tankies are way easier to talk to than Nazis, though I don't really find myself talking to nazis often - just run of the mill bigots. Anyone with consistent standards or ethics is fairly easy to talk to, even if we disagree.
In my personal life I tend to take on more than half of the social costs in some friendships and I probably do the same when arguing with certain types of people. I'm more tolerant than I strictly need to be, but I feel like treating people like that is necessary for me.
The social contract concept is over-used by people who try to make it cover too much. It becomes a one-sided contract of adhesion which you're assumed to have agreed to simply by existing. This, however, is simple reciprocation—it's more like a truce than a contract. It would be unreasonable to expect tolerance from others while refusing to grant the same tolerance to them.
Of course there is no obligation to be intolerant just because the other person is; you are free to make a better choice.
There are levels of tolerance in there. E.g. I'm not gay. I have no interest in men. The idea of being sexual with a man is mildly repulsive to me.
With this, the bare minimum of tolerance is not actively working against the existence and legality of being gay.
Next is the "none of my business" level of tolerance. What happens between 2 consenting adults is down to them.
Above that is acceptance. Gay people have developed their own culture and community. While it's not for me, I recognise that its existence and celebration makes our overall culture more dynamic and interesting. It also provides a lot of happiness to others. Accepting and rolling with that provides a lot of positivity to others, without significant cost to me.
However, if I was approached by a gay guy and propositioned, there is no issue with me turning them down. I try and be polite about it, but being firm isn't being intolerant. (Luckily, most gay guys take being rejected a LOT better than some straight guys do).
Going back to your example. Going up to a black guy and expressing that, while you tolerate them not being a slave, you don't agree with it. This is intolerant, it is an incredibly strong dog whistle of your tolerance is forced.
Conversely, if, during a debate on religion and it's effects, you express your view that you accept people are religious, but don't agree with it, that is better. The context is a debate, and you can explain your reasoning better. It also lacks the dog whistle element that makes it bigoted.
I've found crystallising my morals into words and logic is useful. It both makes it easier to explain, as well as finding holes in my views. My moral framework has advanced significantly over my life. At no point did I think I was immoral, however, I have found significant flaws in my viewpoints. I've also found a lot of biases, which I'm mildly horrified that I ever held.
I'm still far from perfect, but aiming that way, as best I can.