My BIL can neither swim nor ride a bike and he's fine!

Seriously, though, I do get the frustration. My DS (now 7.5, 8 in December) was *extremely* resistant to riding a bike. Extremely. Why did we force it? We live in Holland and you basically *have* to ride a bike to get around. Seriously. It's not only a serious social handicap (kids zoom around on their bikes all day in our neighborhood), but a very serious logistical problem once he was too big to ride on the back of my bike or DH's. So he really just had to. We devised all sorts of methods (the best was running behind him with a long sheet that was wrapped around his body like a rope and we held the ends). He was still late at 5.5 (kids start riding without training wheels at three or four!), but he did it.
Swimming is also a non-negotiable for us because there's so much water around. Plus, after next year, there are some school excursions he'd be excluded from if he didn't have his swim diplomas. Thankfully, he'll take his second diploma test next month and we're through.
DS is not very athletic and is hypermobile.
http://www.skillsforaction.com/?q=joint-hypermobility This makes athletic/physical activities that are easy for most kids harder for DS. It sounds like the boy you nanny for could have something similar going on.
Ultimately, though, it's not your call to make. If it's really impacting your ability to do your job (take all kids swimming, for instance), I think you could talk to the parents. If you read through the link and find that it sounds like the boy, you might also bring that up with the parents (though there's no "cure", DS has gone to a physical therapist and it's helped us all to know and have an explanation for why physical stuff is harder for him). Other than that, I think you have to let it go.