One thing to remember is that, with small children, being the primary caregiver is often a very tactile thing. When there is (finally) some respite (like when kiddo is asleep), touching can be the last thing one wants.
Additionally, the primary caregiver often has to put aside his/her needs/wants when they are "on the job". I, personally, found it more difficult being a SAHP than working outside the home. With the former, I spent entire days with only the child to talk to, do things with/for, etc. When I worked, I could talk to other adults, choose what to have for lunch, go to the bathroom by myself when I wanted/needed.
With our first child, I stayed home (my company was shutting down, and had started laying employees off. I requested to be in the wave of layoffs that coincided with our son's birth). Since my spouse was working, it made sense that I would do nights with the babe as well. All I asked was one night (Fri or Sat) where I could sleep and he would get up, so that I could recharge. Unfortunately, his idea of caring for the child at night was to turn on all the lights, play with the child as loudly as possible, and then bring him to me with "I just can't get him back to sleep!" I quickly learned that it made more sense for me to just get up myself. (And yes, he eventually admitted that he did it on purpose so I'd just get up myself.) I can tell you that it did not make me particularly enthused to provide affection of any sort.
I know another thing I found was that I felt my child actually DID need me. He could not do for himself, so it had to be done for him. My (now ex) husband? Was a grown man and really could have done some things, instead of expecting them to be done for him. I felt that we were supposed to be partners, raising our child together - him by going out to work, me by staying home, but at the end of the day sharing the raising of our child. But it felt like his job ended at 5pm, while mine not only ran 24/7 but the workload increased at 5pm, when I suddenly acquired what seemed to be a second child. It created a lot of resentment.
OP - you asked for people's experiences, and you've received others as well. But somehow, you get nasty when we share OUR experiences with you. Obviously, we are not in your marriage - but you asked for our input/experiences. But you might actually learn something from what we're posting. Your choice.
And to those who have said it's not fair to call OP spoiled? He is the one who described himself as such.