I said "we're just hoping for healthy" or some form of that comment with all four pregnancies - all in response to the "do you hope it's a ___" gender question. It was the worse with dd - we had three boys and people were always saying, "Oh, are you hoping for a girl?" like having a fourth boy was a bad thing. That is what grates my nerves. So, I responded with, "We just hope the baby is healthy" because honestly, we did. Never did dh nor I ever mean that we would not feel as happy with the birth of a not-healthy baby, or an otherwise special needs baby. We were excited to have the baby, period. But, that was my go-to response whenever we were asked about hoping for one gender or the other. As it is, dd was diagnosed with Down syndrome after birth; and we were (and are) still tickled pink (pun intended) that we are lucky enough to be her parents, just like we are lucky enough to by our boys' parents. (And I'm still irked whenever I get the "Oh, you got your girl" comment ... like my boys weren't good enough for us.)
I don't think anyone who says the "I hope it's healthy" type of comment is being insensitive or arrogant. I think it's an honest hope for all of us. We hope that the road our child has in front of him or her will be smooth and even. And I think we all know that it does not matter if the child is born "typical" or born with special needs, that there is no guarantee that their road will be easy. It's just something that we hope for because we love them, no matter how many chromosomes they have, or which special needs they have, or what gender they are born.