I know these feelings, exactly!
When my husband and I first started dating, I told him I didn't think I'd want to have another child, because I'd feel guilty with regard to my first two, who have a "broken home" and developmental delays. How could I make them watch me raise their younger sibling, with an easier life? Much later, when I was married and pregnant and we learned my husband's son would be coming to live with us the day after the baby was born, we both worried how he would wind up feeling about this baby who gets to be with his mother every day, while my step-son will only see his mom a few times a year (she lives across the country).
And you know what? H.o.n.e.s.t.l.y, none of it matters. All 4 of our kids - half-brothers and step-brothers have come to feel and behave like brothers. They love each other, they irritate each other, sometimes they relish chances to be away from each other and have their own experiences, but they're always excited to get back together as the family we really are - not a nuclear, Leave It to Beaver family, but very much family, regardless. And it is an immeasurable gift to children, to have siblings.
Furthermore, each individual child comes into the world with his/her own challenges to face - whether it's poverty, birth defects, learning disabilities, difficult personalities, or tough things that will happen in their lives that they'll have to deal with...or having easy lives and thereby a bigger obligation to help those around them whose lives are harder. We can never fully predict or control what our kids' challenges will be - and there's never a guarantee that siblings' challenges will be meted out equally! On a fundamental level, the fact that your older kids have a "broken home" and your baby won't is similar to one of kid having a disability a sibling being healthy. Whatever the circumstances, you are only responsible to love and guide your children as the individuals they are - not guarantee that their lives are the same.
Your children will feed off your attitudes. Try to teach them there are things to be happy about in their lives and not to focus on the negatives and let that define them. After the baby arrives, let yourself celebrate this addition to your family without guilt. Appreciate and celebrate the family you have, when everyone's together. And give yourself permission to enjoy the special time you and your husband will have alone with this baby. It will not benefit your older children one bit, to think that you are miserable or guilty, while they're away from you. And it certainly won't benefit the baby. It. is. OK. for you to enjoy parenting this little one, even though things didn't turn out quite the way you wanted for the older ones. Let them know there is a happy, loving family for them to come home to, that they're part of.
But right now, you are pregnant and hormonal. Don't stress about how you're going to pull off this positive attitude. Try to have faith that there can be happiness for your family, once the baby comes and that you will be open to it. But it's OK that you don't yet know how all the dynamics will work. You will figure them out as you go along. When you feel like crying, think about what comforts you and give that to yourself. Letting yourself be comforted and distracting yourself from your sadness is not selfish. By taking care of yourself - physically and emotionally - you are taking care of your baby and your family.
Later, perhaps you should explore what - if anything - you can do, to change the older kids' situation at their Dad's. Maybe you need to have more control over their lives and he needs to have less parenting time, if he is truly not meeting their needs while they're with him. But whether or not you can change this will not be entirely under your conrol, either. But that can be tackled after the baby comes, when you're not pregnant.
One woman in a house full of men: my soul mate:
... twin sons:
(HS seniors) ... step-son:
(a sophomore) ... our little man:
(a first grader) ... and there is another female in the house, after all