I intended to keep riding, at least for the first few months, but unfortunately I was so sick that all I wanted to do was throw up the moment I got on my horse.
You might find that when you do ride, you feel uneasy about it, but you might feel fine. I would have ridden my horse, because I know him so well and can predict his behaviour very well. I wouldnt have ridden an unfamiliar horse. I did get a surprise though that once I was pregnant, and even more so once I had my first child, my desire to do dangerous sports diminished. I can't even look at photos of the stuff I used to do (eventing) and you couldn't pay me enough to get me jumping solid fences now.
Maybe when the kids are older, but right now, whenever I ride, visions of past accidents and motherless children haunt me. It doesn't give me much pleasure right now. I"m not interested in plodding out on the trails, only in serious training, so right now I"m just not riding. For a while, I felt mildly - no, majorly - embarrassed, to be one of those wimps who gave up riding when they had kids. But now I'm honest with myself and recognise that you need to listen to your instincts, no matter what your views were prior to having kids.
You might feel fine riding, in which case I'd say to find compromises with your dp, so that he feels validated. But if you don't, listen to yourself. Only you know your comfort levels.