I think it is smart to be careful, but unless you have a baby who had severe lung problems, I don't think you have to avoid everything.
The most important things are feeding your baby breastmilk (by bottle or breast), supporting your family's immune system by eating well, dealing with stress, getting plenty of rest, and taking vitamins/herbs/probiotics as appropriate, along with handwashing. I would ask people to be 100% honest with you about whether or not they are healthy.
Our kids were definitely the most likely candidates to get Zephan sick. I just made an effort to keep them washing hands all the time. And when they did get sick and I thought the baby would be exposed, I made sure I was exposed too, since the baby got the benefit of my immune system creating antibodies that were in my breastmilk.
Be careful, but don't freak out...