I am so sorry for your troubles. I just learned I have PCOS and am also ttc #2, I can't imagine how hard it must be to do it for 4 yrs with no results.
I think you should talk about your feelings with your dh.You are not letting him down, I bet you are a great mother to your dd and a great wife. Tell him what you wrote here about how you feel. I think stress is such a big part of infertility, just sharing with a someone might help.
I am going to throw some ideas out there that you may have already tried, but if you haven't I would try them before going the IVF route. First off, I would stay on the met and maybe up the dose. I'm assuming you are on it for IR, and from what I understand you need to be taking it consistently for it to take effect, sometimes it can take a while. Also try cutting out most carbs and sugars and losing weight ( if you are overweight.) It'll help you ovulate and probably up the chances of IVF being successful.
And don't think there is no hope there is always hope! IVF helps lots of women get pregnant, don't assume it won't work for you. After all, everyone who has gone through it was in the same position as you and it has often worked for them.Good Luck.