It's this way with my husband (who feels terrible he can't comfort our son and has started saying he thinks our son doesn't like him), his grandparents and our sitters. Unfortunately, I have to work outside the home for one full day a week and usually two three-hour jobs a week as well (my husband and I work together, so he can't even have the familiarity of my spouse, even though he cries with him too). When I'm gone, he's unsoothable, and I feel horrible for being gone because I know he's hysterical, and I'm essentially letting him cry it out, against everything I believe.
It's to the point where last night (after picking him up from the grandparents who said he cried for most of the 9 hours he was with them), my husband said it's getting to the point where he's uncomfortable leaving him with people who aren't related to us (for fear of his safety due to someone getting so frustrated at his sobbing). Likewise, I feel awful for my four year old who is getting the short end of the stick. The only time he gets my attention is when I have the little one sleeping (usually on me in a sling or in my arms - otherwise he wakes up and cries within an hour - with the exception of at night when he cosleeps and I'm there) and when he's with another caregiver, their attention is wrapped up in trying to calm the little one, and he's not getting played with by them either.
So, I'm kind of desperate - I have no idea what to do to help him and the others trying to take care of him. Any advice would be so greatly appreciated! Thank you!