This whole thing must really be regional. To me, a party where you bring your own swim cap would be like bringing your own single-person bouncy house to a party. It's just not something most people would have, though obviously a bouncy house would be a bit more expensive! To have to pay $5 for a swim cap that we'd never use again, plus buy a gift for the child, would not be in my budget & I'd decline the invite unless it was a super close friend.
When I lived in NYC all of the pools required everyone (even if you were bald) to wear a swim cap. The membership was super cheap for a local pool ($50/year for me, ds was free), and so we had caps. I think in some urban areas its pretty standard for pools to require caps. I'm not sure why, but I've been to many places that require caps.