Personally, I probably wouldn't (depending on how well I knew the dad). DH is a WAHD, and DD's best friend's dad WAH too (or is the parent that works/stays at home during school holidays/inservice days). So whenever we trade off having the girls for a no-school day, it's pretty much this dad that takes care of them over at BFF's house. I really like BFF Dad, and trust him. Whenever DD hangs out over there she is often gone all day, goes out and about on errands with them, no worries. He was the first non-parent I allowed to take out DD (to the movies) when she was still on crutches after breaking her leg this summer, and he and BFF mom are emergency contacts and have pick up privledges for DD at school. :)
But...I dunno, I would be really really happy to see more SAH Daddies and WAH Daddies around, but even in my pretty progressive, telecommuting friendly area they are rare--or when I have met mamas married to them they get all stressed out explaining that Dad is home until I say , "zOMG how awesome, DH is WAH too isn't that great?!?!" So I get the impression that it is still unusual and might be "weird" in some areas still.
If I had weird feelings for any parent regardless of gender I wouldn't send my kid over unaccompanied. But I don't think my feelings would be tripped just because of a penis. I will admit to other prejudices though--if I know someone is arch conservative religious, it gives me pause because of historical baggage on my part, I am only comfortable with gun owners who feel similarly as I do about safety issues (I do have friends who have firearms that the kids go over to play at their house all the time, but I know these people and have seen their storage and safety measures that they proactively offered, so...). So I do think everyone has their inner squickiness part that isn't always ruled by true true instinct, if that makes sense.