Same dynamic here. And I understand DH's point. He has a very physical job and is pooped at the end of the day, legitimately (as am I of course.)
I can approach this problem in a lot of ways. When I am tired and grumpy (often!), I see it "my" way: he and I are both hard at work all day long. When he comes home I continue to be the primary caregiver of the kids, and continue all of my other work: cooking/cleaning etc. No rest, even at night, as I keep cleaning after bedtime, then change and breastfeed the babe when he awakes crying. Sigh.
The way I try to approach it, though, is to be considerate. The times when I am most considerate and compassionate about how he is feeling (tired, stressed) are usually the times when we all end up having a much nicer evening. He comes home and may relax by himself a while (lucky him!), or may play with the kiddos. We have a nice meal together and a less stressful bedtime. And he may help out with whatever he sees that there is to be done. Or he may not.
It's not fair. I'm kind of over it. I think my DH, like many less enlightened men, just doesn't get it. It is an honest mistake. But trying to get across to him the fatigue and difficulty of the endless SAHM job just comes across as nagnagnag or complaincomplaincomplain. He wants to be a good DH. But he lacks the mindset of a mother: that things just need to be done, no matter how tired or stressed you are. We have a 5 1/2 year old (and a 1 year old), so this isn't new. But he still has this sense of entitlement, even expectation, that he should get to relax if he's tired, have a quiet house to read or watch a movie on the weekend, etc etc. It's pretty funny actually.
So it goes around here. Thankfully he is a loving and good guy who wants, to some degree, to be helpful and all. But I've found that I need to let him find ways to do that on his own. It's just not worth my energy to be bitter about it, or to try to help him be helpful. That, alas, = nag.
Not fair, like a lot of things in life, but how it is. He does work very very hard for us and does not get paid nor recognized sufficiently for all he does. And he bears that. He is great with the kids when he chooses to be (wouldn't that be nice!), which is a lot. Sometimes he'll act like he's doing me a big favor when he hangs out with the kids for a while. The sad part is that he is, as it is helpful to me. But it's not like he should get some sort of "credit", for heaven's sake. I would sure like some "credit" for the 24 hours of every day that I'm on duty, haha!
Oh well. Things are so intense when the kids are so small. We are getting through this. For some it might work to be more pushy (like fleeing the house on a Saturday and leaving all the kids and messes). That would not work for us. We both have our roles. They are not perfectly equal. There are frustrations and rewards on both ends, and plenty that each of us do not understand about the others' experience, and can not. My role exhausts me and frustrates me every day, and also brings great joy. It's hard to explain, but DH just doesn't get it...to him it makes sense that he does his job and I do mine.
I think I'm writing in circles here. Wishing you peace, mama...and wisdom about how to handle your own situation. It sounds like you need to communicate your own frustrations. Just take care to do so in a way that will not bring on defensiveness and anger.
PS: Love the poster above "men are not exactly perceptive". That's for sure!