I've never been all that interested in TV but I did used to watch some. For us, becoming a very low media family went like this:
1- Making a rule that mom and dad never watch TV in front of the kids. It's only something that can happen after they are asleep. Which can rule out a lot because sometimes bedtimes are harder than others!
2- Choosing several or a few shows to follow and watching them in an alternative way. (I watch LOST and The Office personally.) So I download them (for free) and watch them on the computer when we feel like it or have time. That way I'm not beholden to the tv's schedule. And a computer screen feels less invasive.
3- Get rid of any cable. You shouldn't have to pay for TV. It's not worth it. If you want to watch something on cable, chances are you can get it from netflicks or the library if you have a decent library.
4- Get excited about when the kids themselves can watch tv. Make it a really fun time. And just don't allow them to at any other time. So for us, Friday nights are "movie nights" and we get a video from the library. It's a fun trip, there's anticipation. We pop popcorn and make smores and watch together. Also there's been times in our lives (not now though) when on Saturday afternoon when we clean we've allowed an hour of TV and it was also a special time, looked forward to. Any other time we say, "oh it would be fun to watch that, but now is [a school night, family fun day, whatever], we'll save that for movie night!" No negotiating, no sense of deprival, just it's not time for that right now.
So those are some of the steps that moved us to lower and lower media. The other day I was having a very emotional phone call and my kids wanted me and in an attempt to defer them I said, honey, go turn on channel 13 (pbs) and watch for a few minutes till I get off the phone. My kids said, "no mom, that's boring, its not movie night!, play with us." So obviously the low media thing has worked a little too well for us!
Good luck. There seriously is more time without it. And think of alternative things to do with your partner cause at first it can be weird to just sit there and stare at each other like, what do we do now? We clean up together, sometimes show each other clips of things on the internet, read books together, etc... But it's good to have a plan at first or turning the tv back on will be a default move.