Well, this sounds like a marital issue more than a parenting one so I don't know how much that will help things, but here's my 2 cents anyway. While it is totally creepy that there is a known registered sex offender in the area, that's pretty much where it starts and stops- it's creepy. The thing is, there could be (and probably are) sex offenders in many other places that you have been to with your children. I really don't think knowing or not knowing who is on the sex offenders registry changes things all that much. I mean, I guess it gives some information about someone, but as far as how you react, are you going to do something differently than if any other person approached you and your children? I mean, it's not like you would send your kids off with a stranger just because he/she was NOT on the registry. I'm sorry if this is coming across as flippant. I don't mean to minimize the real risk that could be present. I just don't think you and your children are necessarily in any greater danger than in any other situation, assuming you are there to keep your kids safe. It wasn't clear from your post if this person has actually been to the pool or approached you and your children, but I'm guessing not. But obviously, if he has, then that may change my opinion about things.
I am not one to bow to pressure from others, but in this case where the primal instinct to protect is concerned, I think I would want to honor DH's intentions. Talk to him about precautions you can take, and see how he feels.
Does your husband realize that you could have a sex offender living next door - you just don't know because he hasn't been caught? It happens.
I would take the time to talk to him about how to safeguard the kids, but I would not cave on taking the kids if Dad's not there. It is unlikely that the neighbor is going to come running over to molest your kids while you're there.
And if the man acquires a firearm again... it doesn't matter if you're male or female, he could shoot you just the same.
I do think this is a tricky situation because everyone has different levels of comfort & safety. I don't think your DH is being entirely unreasonable, but I think if you're the one that's going to be there, you are the one that most needs to feel comfortable, safe, and competent to protect your kids. Would your DH feel more comfortable with it if you took a self-defense class or something? That might be a compromise so you don't have to wait for your dad to get home...