DS doesn't like my friend's son

I have a neighbor friend who I like to spend time with, and her son is just a couple weeks older than mine. Problem is, my DS doesn't like hers. Her son is a bit on the agressive side, pushes frequently, out of frustration and for no reason at all sometimes. He's never actually hurt DS, but he obviously he doesn't like to be shoved. I've noticed recently that sometimes DS just plain avoids him, like if the other boy comes over to what DS is playing with, DS will just go play with something else, which is really sad to me! My friend tells her son to do gentle touches and tries to remove/distract him from the pushing, but she also has a 4 mo who she's always carrying so she's somewhat limited physically. I try to say things like DS doesn't like it when you do that and try and direct his attenion elsewhere, which my friend seems grateful for, but what can I do to protect my DS? Should I just limit our time with them?

Some background info: my friend and I are not close, we only met about a month or 2 ago so I don't know her that well. What I do know is that she weaned her son with her doctor's advice when he was 10 mo (w/o much of a struggle, I think at that age they're pretty distractable) because she was 6 mo pg at the time and she and her son were losing weight. Then they did CIO : since he didn't nurse to sleep anymore so she has a 18 mo and a 4 mo daughter. Her son obviously needs more attention than he's getting, but she seems to have her hands full nursing her baby frequently and she always is either carrying or slinging her too. Her son is very energetic and they kind of treat him like a "big boy" even though he's only 18 mo. She has her husband hold the baby frequently to give her son "snuggle time" as she calls it, so I know she's trying, but it's still kind of sad for her son because I know how much attention my 18 mo needs from me! She does give her son time-outs, although she doesn't think it's very effective (I don't either).

I like hanging out with this woman, but I am willing to forego that if it would be better for DS. Is it bad for him to be in this situation or am I just over-reacting? Is there anything I can do with either my DS or the other boy to help the situation? The other boy has very good comprehension, he can follow directions well so is there something else I can say to him to curb the shoving? Or is there nothing I can do since I'm not with him all the time? My DH thinks I should avoid spending time with them because he doesn't like that DS steers clear of the other boy.

I would really appreciate any advice! TIA!