Wow that is a really tough position that you are in. I remember two instances like that for me. When I was a junior in high school, my CCD teacher said that having sex after you were engaged but before you were married was fine as long as you were both ok with it. I came home and told my mom I was never going back to CCD when I knew more about the Cathechism than the teacher.
I also had the NFP/BC issue at our Pre-Cana retreat that we did as a requirement before we got married. Now, I am not as 100% Catholic as I was in high school, but it made me so angry that these teacher couples BARELY mentioned NFP, and when they did, they said, "Well, we didn't know about this when we got married, and if we had, we wouldn't have used it anyway." Like it's this unrealistic expectation that no one actually does. Which may be true, as only 5% of people who call themselves practicing Catholics actually use NFP versus other forms of BC.
But you're right...what do you say to that? And it's like you should be able to talk with other Catholics about this, and share tips and struggles and such, but you can't...because hardly anyone is using it! As for me, there are still issues I have with the Catholic Church, but I agree 100% with NFP. DH does not, but that's another story.
The sticky part, as others have pointed out, is what constitutes a grave reason to abstain. Financially unable to care for another child? YES that is a reason; but the question to ask is, would that mean you don't get to go to Disney World, or would it mean that there is a tremendous amount of stress and burden placed on you and your family by the expenses of another child? Do ecological reasons for only having a few kids count? Maybe; that one is super difficult to talk about or get clear answers to. Psychological well-being is also a critical factor. For instance, I would LOVE to have a large family. DH, however, was an only child, and at this point does not feel like he can multi-task well enough to care for more than 2 children. Also, he is a hazardous profession (pilot for the Coast Guard). If something happened to him, would I be able to care for all of the children on my own, in a way that we want to raise our kids with me as a SAHM?
I think that using NFP or not is not up for debate among Catholics, and it is not a matter of conscience. What is a matter of consciences is how and to what extent you use NFP. And that is where the theological becomes personal.