Honestly, not just men, but anyone! I stayed home for a year after #2 was born, and I had a similar experience with my mom (who was supportive of me staying home, but didn't realize how truly stressed I was and why I was perfectly normal for being that way!).
I'm working outside the home now for financial reasons, and even the annoying commute to and from home is an enjoyable break for me now. As I told someone (well everyone), I get paid tons more now and the work is SO much easier.