Kbl, you're in a dangerous situation. Unless someone puts a gun to your head, of course you can escape, so don't use that as an excuse!
What is this weekend group for? How long are both your spouses going to be gone? If your feelings are truly that uncomfortable then don't go.
It's pretty simple, though. If you don't want to have an affair, then don't have an affair. Either get your head on straight or don't repeatedly put yourself in an awkward situation.
I chaperoned for my daughter's week long science camp, and I had a tremendous crush on our counselor (one counselor for every 10 kids). He was very good looking, funny, very well educated, more politically and philosophically in line with me than my husband is. He led us on educational hikes every day for 4 hours, so he was a fit outdoorsy, ecologically minded guy, which I really admire and aspire to. And he was really good with kids, obviously. And the thing that really got me was that after our class was over he was quitting to go to farm school in Idaho, with the ultimate goal of having a small, organic family farm.
I'm married and would never have an affair, so frankly I felt freer to have a crush on him, because I knew I wouldn't feel like I should try for him.
I hope that makes sense. I didn't feel guilty about having a crush, I had that completely compartmentalized from reality. Plus he was almost 10 years younger than I, and I'm simply not his type. You know, married. And we were at camp for only six days. Then it was all over.
It hit me pretty hard. I hadn't had a crush on a guy that intense in a very long time. It got silly towards the end of the week, to the point where I could hardly speak directly to him, and couldn't look him in the eye. (Aw, shucks!