I struggle with letting the guilt of having "me time" go, but, also realize that having that down time makes me a better mom, wife, and a happier person. It's such a hard balance. I've told my kids that "mommies and daddies need time to be alone, just like we have mommy/dd day", and similar statements on a regular basis. I've also told them that I need some time to relax and to go play elsewhere for awhile. You know what? They're okay with it. And, I also believe it all to be true - that taking time for yourself is one of the ways (for me) to actually be a better parent.
Since becoming a parent, I've also struggled with lowering my standards - the list of things to get done is simply, physically impossible. It's okay for the house to be a little messy once in awhile. It's okay for the clothes to get folded the next day. It's okay for dd to miss the occasional birthday party.
Try to let the guilt go. If you don't take of yourself, then how can you take care of others? And yes, there will always be someone with more energy, more resources. Try not to compare - just do what's best for you and your family - even if that means an afternoon of rest!