For us, it was about timing and context. When I broke a bone in my foot when DS was 18 months, we put in a video twice a day so I could get off my feet. We had the same idea as you OP, where we just didn't watch tv in front of DS and hadn't planned to introduce to him for years. *sigh* After I was healed, we somehow just didn't stop the video thing. I guess I felt that on some level it was unfair to introduce it and then take it away... and yes, I liked that 60 minutes to close my eyes during the day.
A couple of months in however, something started to feel very "off." DS was becoming a bit obsessive about "watching." However, I would put something in for him and he'd be watching whilst seeming VERY restless. He would be in and out, his attention span seemed to be waning, just something not right, yk? My gut told me that this was just not a good fit for us, at least not right now. DS wasn't even talking well yet and somehow, this bothered me as well. So, we put the tv away and told DS that we put it away because DH and I wanted to spend more time with him and as a family. In short, the truth. I was worried there would be fall out, but amazingly he was totally even keel about it (he was not quite two I believe). He asked to watch a few times that day and I reminded him why we'd put the tv away. No problems. I also made more time to spend just playing with him for a day or two, to make a smooth transition and it became very clear that DS would rather spend time with MOM anyday, over watching tv. Yes, I missed the ability to slip away a couple of times a day, but without the tv, DS almost overnight became more thoughtfully busy, channeling his energy and building his attention span in a way I didn't think possible in a small child. Our household felt calmer, the equilibrium restored. I thought I would miss him watching and me getting that time but like DS, I didn't miss it at all. I was so glad I listened to my gut.
When DS was nearly three, we brought the tv back. I think we'd moved it into our room (armoir) and DS found it and asked. Somehow, this time it felt ok. DS didn't have that same obsessive/restless thing happening, he was talking beautifully by then, and to date, we have had few if any issues with tv. Like earthmama, we don't subscribe to cable and don't get reception otherwise. DS chooses a few videos from the library each week. I really like this approach not only because it limits commercials and controls content from the standpoint of not being fed whatever is
available on tv, but because it puts DS
totally in control of what
he watches. He (with my supervision--lighter these days then when he was younger

) is his own broadcaster. What I've found is, that he likes to keep his viewing "on topic." He choose videos very much in the same way that he go about choosing books or any other media. When he's into something, he wants whatever he can get his hands on to support that subject. Hence, we watch far more non-fiction than fiction, although he also likes his share of movies and special and situational programming as well.
Lastly, we decided not to "limit" tv viewing. I use the quotes because I believe that people get the idea that if you're not limiting viewing, that your child watches incessantly. However, our experience is just the opposite. By not limiting, we've given DS the ability to limit himself. And he is very self limiting in the area of tv. Admittedly, the other reason for not limiting is that I had no desire to become the "tv police." I find the less I control DS in any fashion, the more he tends to find his own limits. It's the things I've controlled too closely that we run into issues with.

So with that, when DS
is watching, it's because he has something he wants to watch and because he feels like watching. He won't just watch for the sake of watching and he won't watch just anything that's on, only what he's truly interested in. Additionally, we hold a
"don't offer, don't refuse" policy about tv. I think one of the things that felt strange before we pulled the plug is that initially,
we were offering up the tv. We changed this when we reintroduced it as in, tv is fine if you wish to watch, but it's not something I would offer up to you. I believe this has been an important factor in how DS approaches his viewing. He may watch a few hours a day for a few days time if we've brought something home that he's taken with. He just as likely to ignore the tv for days at a time, sometimes weeks. He definitely favors DOING over watching but even in watching, he rarely keeps anything passive. If he likes something he's watched (and this happens often because he careful to choose items that truly appeal to him), he recreates it for himself; draws pictures, sets up a scene, plays it. I like his need to turn an otherwise passive activity into something active. In this way, it now comes from within DS, and from there, the idea has a chance to evolve and grow.
That said, DS has just finished watching "Walking with Dinosaurs" (a fav around here) and so I'm off to play "dominant predator." I really hope I get to be the carnivore this time because DS is getting seriously good at hunting me down and attacking me. He's getting bigger now (nearly 8) and it's starting to hurt when I'm ambushed.

Hope there is something in there that helps, mama. Just my .02 for whatever thats worth. Sorry, it turned into a bit of a ramble.

The best,
Em