How common is it for other dad's to SAH?
My DH watches the kids when they have playdates quite often. Of course, we know quite a few people who have WAH dads like my DH. So really, those families never even batted an eye. My DD's BFF's dad is often the one home when they have a playdate because he WAH more often than BFF's mom. There are a lot of telecommuters here, so men working at home and hanging out with their kids is not seen as weird or girlieman.
If it's not as common in your area, I can understand your DH's discomfort. Though are you really sure that it's because he's afraid of false accusations or because he's freaked out at the thought of supervising even MORE kids or he's worried about rejection from the other kid's parents? Or that he might feel weird entertaining another woman without you there, if the other parent requests (reasonably, IMO) to get to know him better? I do think that in order to protect your DH from nasty rejections, you do need to disclose that your DC have a SAHD.
So it's not a silly fear that he has. If SAH/WAHDs are uncommon in the area, he may very well get some hurtful reactions. I might ask him if he would mind giving it a try once, with someone you know who won't freak out. Or you could encourage him to set up park playdates or something. Once he gets his toe dipped in and people aren't freaking out at him and eyeing him like a potential predator he might be more willing to try a home playdate. But discrimination and prejudice against SAHD is very real in some areas. I think that is slowly starting to change though.