Well, my DH sometimes takes the cake for being an insensitive jerk (a lot of it is him just being clueless and insensitive...he has to be asked everything and things need to be pointed out for him to do something most of the time).
But he has never inferred or said that kids aren't his responsibility, even when I was a SAHM.
Now, he'd work late without considering the impact on me. He never took sick leave to help me out when I was sick. Pretty much all the child rearing activities fell on me unless I asked him to help. But he never outrightly said he wouldn't help because I was a SAHM.
That has to be very, very hard.
Now, that said, I know plenty of SAHMs who are expected to do the bulk of the cleaning, household chores, and errands because they stay at home. I really think that's crap.
In fact, when I was a SAHM, I left some household chores undone for my husband to do because I was busy caring for our child and I didn't want to spend all my time cleaning at the expense of our child.
More on topic to your post, though, I found it very difficult to be a SAHM and still do things on my own like running, or go to a yoga class, or see friends, or whatever.
DH worked late often, and I had zero family support system...no reliable grandparents, etc who helped once in a while. So, I do feel it was easier to pursue my own interests when I worked for a paycheck because I could pay a babysitter, etc.
DH was opposed to having a babysitter when I was a SAHM. It was redundant to him.