I know how you feel. You are not insane.
I have 4 children and I have had 6 pregnancies. We were SURE we were done after #4. I had health issues (chronic), I don't like being pregnant, I don't like giving birth, and I become easily stressed (when there is a newborn) because my support system is lacking (no family help).
However, I've almost taken care of my chronic illness and I'm pretty sure it will never come back. I feel healthier than I've felt in years! Dh and I are always saying that it feels like someone is missing at the dinner table. It is the weirdest feeling! We never took that to mean that we should have another child, until I recently started thinking about another baby. Then it dawned on me that maybe that's why it feels like someone is missing.
Honestly we are happy with our family size. Our income is not large, we don't really have the space for another one. There is really no logical reason to have another. In fact, dh doesn't want another one...but I know he will some day.
So it will eventually become an "issue" since I've been thinking about it and I know he will one day too. He's the one who oohs and aahs over the teeny baby things at the store and refused to get a vasectomy "just in case". It's only a matter of time for him.
Sometimes things don't make sense. Our emotions take over. I hope that you guys can come to a decision on this.