you have every right to be concerned about who your child is with, your the mother and whether people rely on it or not, you have a mother's intuition. that being said, its a fine line deciding to judge someone because of that intuition or just because of their past is difficult to talk about. it seems wrong to just "judge" someone and decide you think they're creepy and not let your child around them. however, what if something DID happen and you didn't trust your gut, you'd never forgive yourself right? so it's a hard thing to say whether or not you are right in your concerns.
if you're still so concerned, file for custody and do your best to get the agreement you're more comfortable with. try to get a good excuse to NOT have the child be able to go down there
on the note of the wife, don't judge her. since you haven't been in that situation, you don't know what its like. i don't so i can't understand either. however, i can imagine that since she knows what its like to be victimized, she would do everything in her power to help someone else if they were victimized, especially a child, especially her granddaughter. i can't speak to the husband, but i think you can trust her