I had a group of three families in a babysitting coop. We met every Friday and even had a standard 3 hr time block. We did sometimes change to another day or time, but only if all three couples agreed. If someone dropped their kids off late, they just forfeited some time, unless they had made arrangements ahead of time. One couple babysat while the other 2 went on a date. The babysitting always happened at the babysitters' home. For a while we were doing 5-8 and so the agreement was that the babysitting couple would feed all the kids.
We chose to do it that early so that it wasn't cutting into bedtimes. It was sometimes a disadvantage, though, because we couldn't go to an evening concert--lots of events start at 7:30 or 8:00.
We got solid commitments from Dads that they would be fully involved in the babysitting. We actually had one dad who would bail on his wife when it was their turn for a date!

But she really badly needed a break anyway so she got a break without her dh.
I had 2 kids, one family had 2 kids and the 3rd family had 4 kids but the 2 oldest were actually quite helpful with taking care of the younger ones.
I suggest discussing ahead what you'll do about cancellations, find out how flexible everyone's schedules are. Discuss discipline--if you don't like timeouts, for example, make sure the other couples are on board with that.
My kids were sometimes overwhelmed with so many kids coming over and wanting to play with their toys--we would go through the toys beforehand and put away anything they didn't want to share.
It was fun and worthwhile. The easiest to organize, though, is just 2 couples trading every week, or each getting 1 date a month or something.