Yes, you do have to draw the line. I was talking with my midwife about it, and she said that it's really important to put our needs first. Our older nurslings still have a need to nurse, but they can begin to understand that mama needs to set limits. Also, somewhere in her book "Mothering Your Nursing Toddler", Norma Bumgarner says something like about nobody liking a martyr. If we're going to nurse two, we have to be happy to do it! We need to find some way to balance everyone's needs. Not easy! It is tough to nurse both at the same time, I get the aversion, too. It's getting better. I've started a count to 10 with DS1, which helps me to feel in control a bit and gives me an end in sight. He seems to accept that well.
Baby is waking now, I hope things get better for you, tillymonster!