I sort of agree it comes across a bit rude, and sets people up to feel uncomfortable. If the invitation says "no gifts," and you bring a gift anyway, you look like you're not respecting the parents' wishes. If the invitation says "no gifts," and you don't bring a gift, there's the risk that most people *will* bring a gift, and you'll look like the jerk that didn't bring a gift.
Also--doesn't it seem sort of rude/weird to tell people they can't give your child a gift? As far as specific types of gifts, I think you just have to let that go as well--this has gotten easier for me as my kids have gotten older. We were invited to a birthday party for one of my preschooler's classmates this year, and the invitation said, "Please no violent toys, or toys that require batteries" as well as including the kid's shoe and clothing size (!!). I was so put off, despite the fact that their values are probably pretty similar to ours (I also hate battery-operated toys and avoid violent ones). Still, my kids have one birthday a year, and I tend to "weed out" toys pretty frequently (for garage sales or charity) as it is, in order to avoid overwhelming clutter (I'm expecting #4 in September, so there are several opportunities for stuff-accumulation throughout the year).
The one "no gift" party I've thrown was for my youngest's 1st birthday, and I said something like (and, this being a first birthday, the guests were all of *our* good friends, so I wasn't quite so worried about etiquette) "Please don't worry about bringing a gift--she won't notice. If you're feeling generous, though, we'd like to put together a little time-capsule for Fiona to open on her 12th birthday, so feel free to bring a card, note, or whatever to add to that."
It was actually a big hit--no one brought traditional gifts, but people were really creative with the things they put in her box--some burnt CDs for her of popular music from now, or of songs they like, many included pictures, most brought cards, someone brought the newspaper from her birthday that year, some sort of "Teen Bop" magazine, and a pack of "silly bandz." It was really fun, everyone seemed to enjoy it, and I think it'll be a really great gift when she's 12.
Anyway, if you really want to avoid traditional gifts, I would give people an alternative, like others mentioned (a charity to donate to, or whatever), but I would really encourage you to just let people do what they will. I, for one, enjoy choosing a gift for a birthday party--especially for a child--and I always sort of cringe when I get a "no gifts" invitation.