You are definitely in a tough spot. My mom isn't quite that bad. But she DOES act like a kid with my daughter, and not a responisble adult. She NEVER tells her no (dd is 2) and lets her do anything she wants to do. She gives her more candy than we allow (we will let her have 1 square off of an organic dark chocolate bar or 3-4 dark chocolate covered blueberries, and she loves it, but Ninna gives her waaaay too much). My mom won't even change her diaper because she doesn't want to upset her. I've left them at my house before to go run errands and have come back to dd wearing a poopy diaper for who knows how long! My mom just will not change her. It's maddening!! She sets no boundries for her at all. My dad, though is pretty good about things, except that he does let her eat and drink things that we don't allow. Like, he'll give her sips of his soda, lets her eat white bread, etc. My mom is not someone that I can just say "stop it or else" to, because she is very emotionally fragile. My dad is another story, I can tell him just exactly what I am thinking. Neither of them touch her inappropriately, but I don't consider bottom rubbing and chest tickling to be a bad thing. I pat my dd on her bottom and tickle her chest, as well as knees, ribs, neck, whatever makes her laugh. Of course, I don't know if the touching in your family is in a different context, or has a different background. In our family, it is completely innocent. My mom has always patted my bottom, still does, and it doesn't bother me. But like I said, it may not be the same type of situation.
Basically, my parents are very loving toward dd and do TRY to go by my rules, they just tend to slip up. It sounds like your mom and stepdad and brother may be a lot worse (my brother-in-law is actually pretty close to your brother, it sounds like, but he's never around, thank God) than my family is. If speaking your mind to them doesn't work, then I don't know what else to tell you. Keeping your kids away from them for their own good is not a bad thing. You are not robbing your kids of a relationship with their grandparents if the relationship is that bad and could be damaging to them. All in all, I'd say that the mental and physical health of your children is a priority over your mom and brother, and stepdad. They are all adults and can make their own decisions, your children need your protection.