Careful with the gender stereo-types. My ds is very very cautious. He always stuck by me. Heck, he's 9 and he still freaks out a bit when I go around into a different aisle in a store and he can't see me. As a 1-2 year old, dd was much more fearless. Ditto for whether fathers or mothers are 'scarier'. It all depends on the situation. Yes, men can in general be scarier, but I suspect my kids are more scared by me when I'm angry than they are by dh when he's angry. I blow up more.
It was essential that I get into the driver's seat as parent. We are very democratic and consensual and all that when in a setting where we CAN. Which is most of the time. But in areas where I need to be the parent to guide him and teach him, he needs to know I'm going to give him the tools he needs to succeed. I mean, it does him no favors if I let him be a wild man wherever we go. Then I'd be too nervous to sign him up for programs and activities because I can't count on him to behave appropriately. He would end up missing out. I need to equip him so he can fit in and succeed wherever he may go. Not like a doormat or a sheep, but with appropriate respect for the people he's with.
Very interesting post -- what strikes me about your experience was that you eventually had to make behavior expectations overt and discuss them. Talking about what he should do, practicing it and then giving feedback (however you chose to do it) was a very effective teaching strategy. My older child needs that kind of teaching. My younger one doesn't. What you found out was that your son needed the direct teaching, and you implemented in it a way that worked for both of you. That's a good example of realizing that your child has needs that were different from what your expectations were, and rising to meet them.