Before we moved back to the US, I spoke English with my children but upon the day we left Sweden, I switched over to Swedish. My son was 2.5 years old and frankly he didn't say a word about it. My husband continued speaking Polish and still does and my son's English is fine. Ironically we prefers Swedish and that's the language he and his little sister speak to each other (despite her being 4 months when we moved).
One thing I noticed was that my Swedish (which I was already fluent in at that point) REALLY improved even more and since we've returned people have commented on that still.
Make sure to find French speaking people in the area you move to and see about playdates/families to help support what you do at home. There is a Swedish school in our area (once a week) and meetup.com
helped us connect with others, message boards and the random chance encounter with other Swedes has helped.
Our situation is similar as well in that my husband could never speak anything but English with me. So we kept that and even though we do speak English in front of the kids, our children have still stuck with Swedish with me and Polish for him. It's all about consistency and creating an oasis of French only (in your case) when at home. Once your child is in school and has more English speaking friends, the real "assault" on the second language comes. The best you can do is ready yourself for that.
While I think it would be nice to speak English with my kids, I've amazed everyone I know and myself that I've given my kids a gift of a third language and that's surpassed any feelings I had about wanting to speak my native language. My kids can now talk to our friends and family in Sweden and enjoy the language, culture, and hopefully gained opportunities down the road.
Best of luck whatever you do, but I wanted to let you know that it can work out really well!