This is a calculated risk that can be calculated out by having only adult women care for my child, thus leaving me with fewer fears. There are still plenty of risks out there, but this is one that is far less likely since I made this calculated decision. It is not unevaluated, in my case. I was never sexually abused, but I know plenty of women who were--all by males. My Dd has plenty of interactions with men and boys, and the decision not to have male caretakers is something she is oblivious to, so she won't have any problems coming from having been more protected in this way.
This discussion seems to be another issue of perceived threat vs. actual threat. If 1-3% of men are child molesters, the chances are very small of your child's male care provider being a molester even if you chose a care provider totally randomly. Using your "gut" as well as other tools to educate yourself about a male care provider (e.g. interviews, references, background checks) should reduce that chance considerably.
Yes, even after all of that, you are taking a calculated risk when putting your child in ANYBODY'S care. Your child might be with a caretaker, teacher, clergyperson, bus driver, etc. that is a molester... or physically abusive... emotionally abusive... a substance abuser... mentally ill... or somebody who could in many other ways be dangerous to your child. Hopefully NOT, but there is always that chance when you leave him/her with anybody but yourself.
I don't think that simply keeping your child away from all men is going to provide the protection you think you are giving. You may FEEL that way, but the statistics don't (IMO) bear out the perceived threat vs. actual threat.