Since everyone's body is different, I think it can be a factor for some women. Not all, though! I think if you're charting, and you know you're ovulating, then it's probably not a factor. We've been ttc for 2 years, and I'm still nursing my almost 4yo. One doctor told me to stop immediately (when ds was 2.5). Ummm, NO. The last doctor thought there was a possibility, but he thought it was slim and didn't think it was worth weaning to find out, since there wouldn't be any going back then. And after I explained how much total time I was still nursing (3 and 4yo and very efficient nursers!), he was sure it couldn't be a factor.
I've heard that night nursing has more of an impact on your fertility than any other time of day, so if you think it could be an issue for you, you might try some night weaning or limits.
Kristen