If you believe the children will be safe and their needs will be provided for while you're gone, definitely leave him alone with them for a few hours (or a whole day, depending on how mad you are ). If you genuinely don't think he's capable or that he'll be passive aggressive and let the children go neglected all day, then hire a sitter and let him know you'll be back in a few hours. But do something for yourself, mama. You deserve a life too.
He sounds very disrespectful of your role as a mother and wife, and of your right to have time to yourself. If he's not willing to be a father to his children, he'd better be willing to pay for someone to come and relieve you once in awhile. But he doesn't get to tell you that you have to do it all, and he definitely doesn't get to tell you you're crazy for needing a break. The next time he says something that insensitive and awful, point out that he's being emotionally abusive. Because he is.