For me it would help to have examples of what kinds of things you are asked to do that you don't want to do, but I agree with the general advice you've gotten: "I'm going to have to pass" or "I'm sorry but I can't" really are enough, you don't owe anyone an explanation.
And I think that's a good place to start.
I do also think though, that as time goes on and hopefully that gets easier/more comfortable/less awkward to say, it is worth it to experiment with telling more of the truth. Example: my husband has recently made friends with another father of a child who lives right in our neighborhood with his family. On the surface we have a lot in common with them: about the same age, one kid each about the same age, and we live nearby. And I also really like the husband. But the wife and their child... not my favorite people. Their son is so negative, which I really believe he gets from having no boundaries set for him at all and getting what he wants by whining all the time, after spending time with them our daughter starts to mimic some of those whining behaviors and pouty faces - things she rarely does otherwise. We nip it in the bud pronto, but it makes me not want to spend much time around them myself, especially since my free time is so limited.
So the wife has asked me several times to either go with her somewhere (she never wants to bring her son and I like to bring my daughter most places, so right there we want different things cuz she wants to do things you can't do with kids) or come over with her son, and I pretty much always not only say no but say a little bit about why "We're on our way out to visit family" (if it's true); "We're having a mommy-daughter day and want to just hang out the two of us" (always a good backup if we don't have other plans); or "I'm so tired, not up for socializing" (almost always true in this case!).
Not only should you never feel any shame for just not being up for something, I think it's healthy and useful to be able to let people know a little about why. The people who try to argue with you or change your mind... probably not worth spending much time with anyway.