Two years ago, almost all of our activities outside of school were with other families with special needs. My kids weren't ready for other types of activities (it just caused them stress), and I was fed up with having to explain things wherever we went. Now, my kids are ready for more. My 6-year old with autism started weekly Religious Education classes last fall. I stayed by his side for the first few weeks, then faded into the background. Now, I wait in the hall so I'm available if I'm needed, but I haven't been needed yet. He loves it and he is accepted by the other kids. We're starting to do more mainstream activities with all of the kids now.
I do talk to the instructors/coaches first. I let them know the situation, the limitations, and the accomodations. So far, everyone has been welcoming. If I am the best helper for my child, then I help, and if I'm not, then I stand back. I don't worry about the other parents unless there is a reason they need to know -- the need to explain was mine because the "right" to know wasn't theirs.
If you feel good about the activity and think your son would benefit, it's worth a try. If it doesn't work, you can always pull out. And if it does, what a great experience!