We are a TV free household, but we listen to a lot of audio books. I started using them with my daughter just before she turned two. Mostly, I put them on in the car - but as we live rurally that can easily equal an hour a day of audio book content. Quite often when we get home from an excursion my daughter wants to stay in the car for another ten or fifteen minutes to hear more of the story!
It used to worry me a bit, but I've come to see the audiobooks as a huge benefit to my daughter. I think that it's really helped with her ability to listen and appreciate longer stories. She turned five just last week and we've been spending this summer listening to a lot of E.B. White and Roald Dahl. It's amazing to see how the vocabulary from what is currently on high rotation gets absorbed into her speech.
I see that your son is about six months younger than my daughter. Have you tried the "Nate the Great" series? "Pippi Longstocking", Bill Harley's collections of stories, and the "Geronimo Stilton" books - although not great literature by any stretch - were also huge with my daughter at around your son's age.