Well, I live in a state that just outlawed smoking in restaurants last year; however, when I was pg with my son, smoking was still allowed, and I just did not go to gatherings with friends. I missed doing that but I do not want ANY smoke entering my system. I just told my friends that I wasn't feeling well, or that I had other plans until I made the pregnancy public.
As for visiting family, I am sorry if this sounds harsh, but I would not visit. Smoke clings to clothes, walls, everything....just b/c he smokes outside when you are there, that does not mean anything. I would not visit. Rather, I'd have them visit me, and say "NO smoking AT ALL." Not to mention - you don't want to be around any smoke after the baby comes either. Taking in smoke at an early age can causes all kinds of problems later on, including asthma.