I hated being the boss! Despite perfectly good nanny experiences, putting the kids in daycare was such a relief.
I don't feel like a pushover generally but my job has enough strife, more at home kills me and results in me being a super softy. (Typical I bet.) For me, picking a nanny who wouldn't need pushing was huge. I didn't end up picking the most proactive person but the characteristics I honed in on thinking they would save me emotional grief turned out to be very valuable. It was very important to me to find someone very mild-mannered.
I never did performance reviews but did revisit the contract annually to update the tasks based on kid development/needs. If there was an ongoing issue I mentioned it in person on an "up" day for both of us, or my husband would in the morning. We tried to give the nanny space and a sense of control over her environment and respect for her skills/expertise, and only offered feedback if she was straying too far from our vision, which was not often.
I'm not sure if that helps but I would say if you've got the right person it seems like maintaining a healthy relationship should come with some ease.