I took DD out to a restaurant for a friend's going away party when she was 3 days old....in a car seat, with people holding her. It was May, though, so not flu season or anything. I probably would have anyway, actually. I think I'm out of the ordinary in that regard.
If you labor before birth (vs. a scheduled c-section with no labor) baby gets tons of antibodies from you and is about as immune to everything as they're going to be for the next 6 months. It's as the baby gets older that they become more and more susceptible to things, because your immunity is wearing off and no longer protecting them. The real risk for them at the newborn stage is catching something you have no immunity to, but if that's your fear then YOU shouldn't be out either, because YOU might catch it and give it to the baby.