If you do go on the Pill, please request the "mini pill" that contains only progestin. Estrogen (half of regular Pills or triphasic Pills) may decimate your supply. The Mini Pill will stop you from ovulating so the pain won't be there.
You can also treat the pain, I also get really sore (nursing or not) when I ovulate, sometimes it only lasts a few days, sometimes it may last until I get my period. But, I have a history of Endoemtriosis, and if you have pain in your ovary duing ovulaing or new pain during sexual relations, you may want to see your OB to make sure you don't have Endometriosis or even an Ovarian Cyst. Ovarian Cysts can cause ovary and nipple pain and can cause you to throw LOTS of excess hormones into your blood making you extra miserable. Your OB will probably be able to see the Cysts on an ultra sound, but will need further evalution to find out if it is endometrisis. Endometriosis doesn't show up on ultrasounds because the "implants" of endometrial tissue are nearly the same density of your uterus and intestines, so other means will need to make a diagnosis.
Of course, using Motrin (ibuprofen) or Aleve (naproxen sodium) can really help with the pain. I find naproxen sodium more useful for this type of pain (especially if the uterus or ovaries are also tender) but you can experiment to find what works for you. Neither will cause any problems with breastfeeding and are acceptable by the American Academy of Pediatrics for use in Breastfeeding Mothers.