I honestly think it depends on the person-- male or female. Some are good, some are not. some are scary, some are not. surely this male nanny would have references? Or at least a reputation through others who've used him? (I guess that's references, no? )
If he was reputable and trustworthy, I'd have no problem whatsoever.