I've been in a somewhat similar situation of knowing I have to work and wishing to be a SAHM. Luckily I'm quitting my job next week, but have been working since my maternity leave was over at 8 weeks PP (DS will be 11 months when I stop working).
I can't really address your partner's second concern, because I don't know your financial situation. I know a lot of moms here on MDC have told me that if I really wanted to be a SAHM, I just had to go for it and it would work out. I doubt that would have worked for us since I was the sole income earner. Luckily DH got a great job offer, so he gets to go back to work (he has been a SAHD) and I get to stop. But it took him getting a good enough paying job to make it happen for us financially.
Regarding the first reason he gave you, it's not really his position to say that to you. Honestly it seems like a cop-out to support his wishes to have you keep working, maybe because he's worried about it financially. While it's valid to be worried about finances, it doesn't have anything to do with how you'll feel as a SAHM. You say that this is something that you dream about, so I think that you really want it to happen. If you find yourself in need of adult responsibilities and connections, you could always take on a different part time job that's not related to your career, a volunteer job, or go back to work full time. You aren't stuck in that decision, but you'll only know if you try it.
As for us, DH is torn between being worried about finances and and wanting DS to be home with us and not in day care. We have been working for months to find a situation that works financially, so we decided that I will try it out for 6 months and we'll see how it works. If we can't afford it, then I'll find another job and we'll go the daycare route. DH was also worried about my career (I have a good job now, what if I can't get a similar good job when I decide to go back to work?), but ultimately I told him that it was my choice to put that on the line because of how much I want to be a SAHM, and I was willing to take that chance.
Sorry about all the ramblings. I wish you luck in your decision!