We had a similar situation in 1:50 miniature a few years ago. It was just one kid, but things did not always go well with her.
I echo what the others have said: be firm but kind with the kids, like you did in the backyard. Don't feel you need to give them excuses (unless there is really an excuse and your kid really does want to play when the excuse is over, like he's in the bathtub but when he gets out he'll come find them, that kind of thing). They do not need a reason, it's just "not a good time." If they want to know when, find some nice way to tell them you will not answer questions about the future. Coach your son about how he can set limits with them so he can play without you having to be his bodyguard. This doesn't have to be a complicated relationship with them, and if the parents are odd and defensive, which sounds like it's got to be among the possibilities if their kids are badly behaved and poorly supervised, it's probably better to be as simple as possible with what you tell the kids.
Continue to be kind, do invite them over if your kid ever wants them over.
Think about a fast-growing hedge, or a stockade fence. Remember that the sidewalk is common property and will always be a free-for-all.
Edited to add: The trampoline is probably a huge draw. Finding a way so that they can't see it, like with a hedge or fence, may be helpful. We were the ones who had set up our high wall, and it was to keep our climbing-then-roaming dog in, but it also served well to shield us from the temptation of our neighbor's pool, which came along years after the dog. We loved those neighbors, but I'm glad our kids weren't always pestering them to get in their pool (or on their trampoline -they had one of those too).
Last edited by Letitia; 05-12-2017 at 12:33 PM.