This is us...and pretty much all of our friends so I totally get it. One set of friends in particular are super harsh physically and verbally with their 4 kids (she spanked and verbally humiliated the 3yo while potty training because she had an accident). My 26mo DD is a bit uneasy around 3 of the 4 - only one she gravitates towards is the 2nd youngest, a boy, who's 8, because he is very reserved and super gentle and attentive to her. I basically limit interactions with them because of how violent things can get and the confusion on DD's face when it happens is enough for me to call it a day. If you can limit interactions to one on one with them at your home when it's less chaotic it might help. I notice these 4 in particular are night and day from my house to theirs. In my home they fight much less and often look to me more for direction. I will say they seem to appreciate my kind approach much more and that is why they gravitate towards me vs their own parents, pretty sad when you think about it. But even if that's the only real positive exposure they get, it's certainly better than nothing. You may find the other parents commenting on how calm your LO is, which is a great way to bring up the topic in conversation. I have had people point out how well DD listens and they might go on about how they yell and get no response and I have found great response to be "Yah, I've tried that a bit, but you know what, it never felt right for me and it didn't work well either. It seemed to escalate things more. I find if I listen to her and we go about it calmly, she responds quicker. And hey, I can remember being a kid getting yelled at, then sneaking behind my parent's backs to hide and trying to avoid the consequences...I dunno, I guess I just want to avoid that with my kid, we'll see if it works!" It keeps it casual, doesn't make it seem like you are condemning them for their behavior, but at the same time lets them know that maybe they could change what they are doing and get a better outcome.
Oh and as for your LO, I have his same problem after we've played with the "wild ones" - she gets a bit pushy (because that's how they are) and so we talk about why it's not nice to push and how in our family we treat people politely even if they re not polite to us. She's 2 so I keep it simple and really try to intervene when it happens. As she gets older I'll be able to have more in depth conversations but for now the avoidance is unfortunately our best option with some.