That sounds awful. I wouldn't go back. Are there enough families in your town that you could start a less-structured co-op? Even with just another couple of children?
That sort of co-op certainly wouldn't have been for us. See if you can set something up for yourself. It doesn't have to be a learning co-op: even just a regular play-date, sometimes combined with a craft or field trip, or afternoon at the park or the pool. Stuff like that tends to provide more social time, and more opportunity for casual parental support, than structured classes.
You don't have to turn homeschooling into a group activity. If you want to, the co-ops are a nice addition, but if they don't work for you, don't worry about it. If you want to run into other homeschooling people just to know some, I have good luck at the library during the day. I'm always amazed at the new (to me) faces I see.
Personally, I'm much happier going my own way with homeschooling.
I wouldn't do that co-op either. However, it is how most the co-ops around here are ran. Additionally, there are lots of social groups too for homeschoolers in our area. There is a yahoo group that people post on for meet ups and there are some regular eventst that started out on that group. My dd goes to a regularly scheduled "bookclub" that is mostly a social gathering. She LOVES it. There are also park days each Wednesday. Not everyone hits all those, but it did create a spin off for younger kids. Also, someone started a "lego club". The kids just gather to play/build with legos. I hear it is going really well. Now, many of the parents that attend these groups are also part of a co-op, but not all.
Suggestions: ask one of the mothers at the co-op if there are any gatherings that are open to the hs community in general (not just for co op members). If yes, try that out. If no, start one. Send a flyer to the parents at the co op and start a facebook page or yahoo group for homeschoolers of your area. Post at the library. Go to the library during school hours and see who else is there with kids. Ask the librarian, sometimes they know. I wouldn't give up on the idea of homeschool simply because you haven't found a circle of people yet. And as for the neighsayers, poo on them! Instead of "yes we hs and we love our co op", just change to "yes we hs and we love it!". If they raise concerns, remind them that he can enroll in school at any time and you are taking it a year at a time.
Do you belong to a local homeschool listserv? That is a great way to find out what other opportunities there are in your area and maybe meet other homeschooling families nearby.
I agree with the suggestions that you should try to find opportunities to just get together and play with other homeschoolers, but while you're doing that, I would be very careful not to complain/criticize the coop to anyone local. It will get back to the organizers if you do, and it will not make your life better.
This fall I worked with another mom in my area to start our own co-op, and the preschool class is mostly play, and the kindergarten class has a lesson, but also has free play time. I needed a co-op that worked for me, I have 3 kids at 3 different ages, and I wanted a group I could afford and we could all attend. So I helped create it. That said- when I only had a 5 year old and a baby, I found a couple of homeschool friends and set up a very simple co-op, which was really a play date with 2 short 15 minute lessons taught by me and another mom, she did music and I did simple Spanish lessons, we also did our own holiday parties.
So if your area has homeschool park days or anything where you can meet another mom- it isn't that hard to create what you want.
I find that finding co-ops with like-minded families is challenging and I live in the burbs of a big city! We have 2 homeschool social groups that we occassionally attend. We enroll the kids in lots of classes in the evenings and weekends that they like. This way, they (a) have social time with peers, (b) have fun in a slightly structured to very structured environment, and (c) are meeting kids that often share the same interests, like art, ballet, soccer or karate.
It can get expensive but it works better for us than trying to do a co-op (recreates the class, too structured, we're secular and most co-ops aren't).
Don't give up on starting your own social group. One of the social groups we belong to started with a homeschool mom who went to the local park every Friday at 11:00AM and told other homeschool families (through Yahoo group, twitter, etc) that this was her "group" and join her if you want. After about 6 months, it caught on and now it's the biggest homeschool social group in the arear. Very inclusive and very unstructured. Just lots of play time most Fridays at the park!