It never bothered ours any. I've had my medium fuzzi bunz for more than four years, and they've been used as swim diapers countless times, and yeah they're worn out a bit, but four years is going to wear a diaper out, so I don't think it's the chlorine. That rinses out easily. There's chlorine in my city water supply, so my diapers must be exposed to it every time they're washed, anyway.
A hint-- if you're using a pocket diaper or diaper cover for a swim diaper, just use one that's a bit too small for baby normally. Without a diaper inside, it won't be too small, and you'll get a snugger fit around the legs and waist. No need to buy some specialized swim diaper at all.
Some people have talked about the pocket filling up with water and weighting baby down, but it's not like you're going to leave a baby alone in the water, so you'll be right there to snatch baby up if that ever did happen. And in my experience, it doesn't happen.