Trusting or not trusting someone based on their gender could turn out to be very, very dangerous.
If you're willing to assess your own child's risk by this means, go ahead. I'm going to choose to distrust women and men equally with my son's life.
No matter how careful parents are, weird, unexpected stuff happens. You turn around and your child has wandered off. What then?