I think video games are great and can be so many things - from engaging in problem-solving to being really creative and artistic. I think video games are, in large part, misunderstood and the good is unfairly lumped with the bad. The people who really dislike video games are usually those who have little experience with them. I have great memories of solving puzzles in adventure games with my family - all four of us, when I was a kid, around the computer or just in the room and offering suggestions on how to solve a riddle or a puzzle.
There are some amazing games out there, many of which encourage really positive skills like hand-eye coordination, creativity, critical thinking, problem solving, etc. Sure there's trash - but there are trashy books, trashy movies, trashy internet. There are plenty of opportunities for making video games a really positive experience for everyone. And, of course, as with anything, moderation is key. People who have addictions are a different matter and need to deal with those issues but in the same way I wouldn't warn people off a glass of wine because I know there are alcoholics, I wouldn't warn someone off video games because some people can develop an unhealthy relationship with it.
That said, no one should do something just because everyone else does it. I think if you dislike video games because you've never really given them a try (I mean really, how many games have you played start to finish; how many unusual or unique titles have you even tried?), then that's a shame. But your option, of course. And of course you don't have to buy any for your kids. We don't have cable TV or antenna (just Netflix and whatever we might see fit to buy on DVD, which isn't much), and I don't feel like the kids are missing out on much (other than the commercials, thank goodness!). So I know what it's like to be a bit "odd" in that respect, and hold an opinion different from the majority. I don't feel any pressure to run out and buy cable. Ultimately, it's a personal decision and I don't care what other people think.
If at some point you decide to let your kids play video games, I would strongly recommend that you learn more about them yourself and figure out what types of games - and what specific games - are appropriate, fun, and foster positive skills. It's not hard to avoid violent, negative, or stupid/boring games if you know what you're looking at.
FWIW, I would cut out TV/movies altogether long before I'd cut out video games.