When a person has young children at home, it is emotionally - but not at all practically - possible to love them, separate from their children. Your BF and his kids are a package deal, just as you are a package deal with your children. You would be wrong, to bring someone into your children's home who doesn't like them. It is just as wrong for him to do so.
He may not see this. He may want you there because he loves you and may be turning a blind eye. Or he may feel, "It's only every other weekend". If so, do you REALLY want to be with a man who makes his GF more important than his kids? You shouldn't.
It doesn't matter if you're right and his son IS bad. (Although, with a 6-y-o, I don't really believe that's true. I believe he may be poorly parented, or have parents/caregivers who don't understand his needs.) Bad or good, he is six and he deserves to be the most important thing in the world, to both of his parents. He should not have to live - even EOW - with an adult who does not like him and thinks he's inherently bad.
And you should not have to pretend you think otherwise - nor inflict this child on your own children - for the sake of loving your BF. You're not married. Raising (or "fixing") his son is not your responsibility. Your only responsibility here is to be mature enough to put the children first and not do what's wrong for them, because you can't stand breaking up.
Besides, long-term, how do you see this working out? Do you imagine he will change and become a better parent? Do you imagine it will eventually bother you less, that he slacks off in disciplining the kids and makes you the bad guy? What if you two have a kid together? You'll be dealing with the same problems, but you won't have the same freedom to walk away.
Based on how you worded your own post, I think you suspect this is not going to work out. Listen to your wisdom, more than your need to feel like you're not alone. The first will help you be strong and find a better situation. The second will keep making you feel weaker and more dependent. Do the right thing for yourself and the children who are looking to you as their example.