Totally nothing to do with gender but I would find it a bit uncomfortable to send a kid home with another parent I don't know, without ever having been to their house. Maybe if the child was Grade 4 or older, but not for a little one. My daughter is 4 and in pre-kindergarten and has been talking about playing with classmates after school, but I think I would want to accompany her - at least to see her in the door and stay for 5 minutes of chit-chat before leaving her. I would also probably leave it open ended how long we would be able to stay that day so that you can leave early if there feels like a reason to. Having said that, I know that in the natural way things happen, there are some of my daughter's friends who invariably end up using our house as the meet-up place, and very rarely play at the other kids' place, and then there are other friends where my daughter would always go to their place rather than them come to ours - it just seems to happen that way. It could be that for the first several times, you could have the play date at your place, and then maybe offer to drive the other child home, and let the kids have a little play once they get there - giving you a chance to scope the place out, but without any expectation that you would be leaving your kid there that time.
Then again, I don't think I would be insulted if someone said "hey, that's great that you are so able to let your daughter go like that - I think I'm building up to it but am not quite there yet - mind if I come with her for the first couple of visits?" If someone said that to me, I'd totally get it - every parent has different thresholds. Another possibility is that since the gender issue did become such a big thing here, it could be that he's been rebuffed in the past if he's tried to offer to host moms who thought that was weird. Not to paint with a broad brush, but a lot of guys are less into reading into everything and more into just talking about it, so maybe that's the best way to go!