DH & I listen to such a huge variety of music that we pretty much decide on a song-by-song basis. Our son (4.5) LOVES music of all kinds, but his preference runs toward classic rock with lots of guitar and good drum beats. We made him his own playlist on our iTunes, and when he hears a song he likes he asks us to put it on his playlist. Sometimes he likes to click around our full collection and find new stuff, but we're always listening with him and he is fine with it if we ask him to skip a song because it might be scary or have bad words in it. 90% of the time he listens to his own playlist, which are all "approved" songs: ABBA, Beastie Boys, The Beatles, Billy Idol, Bob Marley, Cake, CCR, Grateful Dead, Joan Jett, Led Zeppelin, Michael Jackson, Prince, Queen, The Ramones, U2, Violent Femmes (to name just a few!) I can probably think of at least one song (in some cases many!) by each of these artists that I would prefer he not listen to until he's older.
I know he listens to the lyrics because he will sometimes ask me what a song is about - but there are other times when he is singing to himself a song that I *thought* he knew the lyrics to, and I realize that he has just memorized the sounds and not the actual words. Kind of like the way opera singers learn to sing in Italian. We (DS & I) have also had some very cool conversations about the emotions we feel when we hear certain songs. One in particular I remember was about the song With or Without You by U2, which is one of his very favorite songs. He said that he could tell that Bono was sad when he sang that song, and he had this whole theory about why that was, but I remember being amazed and very moved that a 4-year-old could empathize so well with the emotions he heard in music.
We don't have a whole lot of kid-specific music because DS just isn't that into it, but we do really like They Might Be Giants, as well as the "Snack Time" CD by The Barenaked Ladies.