I still take my prenatal vitamins, but I don't supplement the baby.
The quality of your milk has little to do with your diet. I posted this in one other thread but i can't remember which one...
http://www.kellymom.com/nutrition/mom/mom-diet.html
"Do I need to maintain a perfect diet while breastfeeding?
The short answer to this question is NO – you do not need to maintain a perfect diet in order to provide quality milk for your baby. In fact, research tells us that the quality of a mother’s diet has little influence on her milk. Nature is very forgiving – mother’s milk is designed to provide for and protect baby even in times of hardship and famine. A poor diet is more likely to affect the mother than her breastfed baby."
...so in short, yes, you're right. Baby will get what he/she needs and we will be the one who lacks. That's why it's important to try to eat healthy, take a vitamin if you can't/don't, and especially get enough calcium, etc.
Follow Mothering