That's a funny question! I only bring it up when otherwise i'd be lying. Like if someone is going on about how obsessed their child is with some character and they ask me if mine is, I 'll say,"No we don't have a tv" It has actually opened up some good conversations. I'm sure someone thinks I am annoying!!
But I HAVE known some holier-than-thou TV-Free types and (just like holier-than-though *insert any other choice here* types) it drives me a bit batty. I know that it's partially a sensitivity issue on the part of the TV-having-set (not wanting to feel that they're being criticized or looked down up on because they *gasp* allow their kids to watch TV), but, as in anything else, some people DO get a little... um... uppity about it.
My POV on the matter is that if, in the normal course of the conversation, it would be awkward to just blurt out "We watch TV!" , it's probably awkward to blurt out, "We don't watch TV!" If it's a natural part of the conversation, I certainly don't hide it, but it's not something that tends to come up all that often.
The fact that you're even concerned about being annoying probably means you won't be.