Most of the time for me, I just use the term "home birth" and hope that settles it. Strangely though, the majority of people go on to ask if there was a midwife there! I always find that odd - I figured it would be an unspoken assumption. I'm not comfortable with straight lying, and I've done it twice now already, so I do go ahead and answer the questions, but depending on who it is, I tend to hedge and say as little as possible.
It usually ends with them telling me their hospital nightmare stories, and then making me feel like a freak show, as a PP said above. It's in a good way, with the whole "brave" and "superwoman" nonsense, but still feels like a freak show nonetheless.