I realize you posted this a while ago, but I just saw it and feel compelled to respond.
What you describe with your husband is how my ex was. We had twins who were born super-early, with major health complications. Although I'd never intended to be a SAHM, once I finally got them home from the hospital I knew I had to care for them myself and any complications that created in other areas of my life, be damned. Heck, with all their special needs, we couldn't have afforded to pay strangers to care for them full-time, anyway! My ex worked full-time, outside the home and had a well-paying job, with great insurance. We had everything we needed, on his income. But, even though being left alone with our kids for a mere hour seemed like hell to him, because they were so frighteningly fragile (and demanding and nonverbal, like any babies!), he just could not get past the idea that I was "doing nothing" because I wasn't working outside the home and earning an income; and that there was something embarrassing about my ability to stay home and not hate it - I must have no drive or ambition. And naturally, if absolutely everything was not spotless and perfectly in order, in our home and with our finances, that was further proof that I "just sat on my a** doing nothing all day" (which he said more than once). As though physically caring for our children, interacting with and stimulating them, overseeing all their medications, treatments and therapies around the clock and carting them everywhere - including weekly doctor's appts. - with apnea monitors and oxygen tanks attached... was nothing, if I didn't manage to vacuum and wash the dishes, before he walked in the door!
He slowly lost all respect for me, and showed it. For a while, I felt terrible about myself and kept thinking, "If I can just do more. If I can just keep things cleaner, or quit taking naps and use that time for paperwork and financial stuff. If everything about our life were perfect when he got home, then he'd realize I'm doing something of value with my time." But that didn't last long. I was exhausted and I KNEW there was value in what I was doing. HE may not have understood the importance of me playing with our kids, but he was wrong. When he finally told me (when the twins were 2) that he'd signed a lease on an apartment and was leaving, I felt surprisingly OK with it. It was a big relief.
Maybe I should have left him, but in retrospect, waiting until he did it put him in the position of feeling guilty (whereas if I'd left him, he might have felt rejected and angry). I think this made him more reasonable about discussing financial and parenting-time arrangements, instead of trying to use those things to be punitive, as so many people are, during divorce. This dynamic may be something for you to consider, with your husband. If he's threatened to leave, but hasn't, some sort of guilt is making him hesitate. If you can stand staying with him longer, you might let that play itself out. If he gets to the point of actually leaving, but some tiny, inner voice is still telling him he's selfish and unreasonable, he may try to make amends for that by not fighting you about reasonable custody and child support. I'm not saying be manipulative and nasty toward him. He's still your kids' dad. But there's no need to go about this in a way that makes things even harder, for yourself.
In my case, my ex and I were both very young (early twenties). He had no experience with kids, whereas I had a lot and was much less intimidated by our kids' needs. After a couple prickly years, we came back around to being friends and somewhere along the way, he seems to have developed a lot of respect for me, as a mother. I don't know if that's due to input from his family and friends that changed his attitude, or the fact that as the kids have grown older (and verbal) he's become more confident in his own parenting skills and spends much more time with them and now realizes the work involved in raising kids, and its importance...?? Interestingly, he's now married to a woman who has never been interested in working outside the home. They have a toddler and she wants 3 more! They do have a beautiful, spotless home, but he pays for a lot of expensive, professional help with that and by no means expects her to do it all by herself - even before she had a baby to care for. In a very small, subtle, Freudian way, I look at the lifestyle he's chosen with his wife as a tiny, quiet statement to me: "I'm sorry. Perhaps you were right."
In your case, it sounds like your husband has continued with the same attitude too long to expect it to change. It is NOT your fault. I'm not telling you to leave your children's father and break up their home. But if you do - or if he leaves you and you feel relieved and ready to find someone nicer - there is no shame in it, for you. Remind yourself of that, because he'll want you to think there is.
My current husband does get annoyed sometimes, if our house (with 4 boys in it, now) gets messy (and boy, it really does, sometimes)! But I tell him he's welcome to do the dishes or run a load of laundry and help. And he does. And he never tells me staying home and raising our kids is nothing. Never. And you don't deserve for anyone to ever say it to you, either.