My first was 5lbs15oz at 39 weeks and gained VERY slowly, at 5 she is barely 30lbs! It was such a huge stress and the constant comments about how small she was really hurt! I feel you mama! After much pressure i broke down to the pediatrician and she had tons of tests her first three years and she is totally healthy! One of my best friends at the time had a 10+lbs baby who grew like crazy after birth and the comments she got were equally hurtful, no matter what you worry and wonder!
Kendra was 8lbs1oz at birth and almost 11lbs at just over nine weeks and I have not really had any comments on her size and I like it that way! I do not understand why everyone always wants to comment on size.
I think diet can have a little to do with it but more so genetics, at least in my case. My first I ate pretty healthy and she was less than 5lbs, my second I did not eat that great and was not very active and she was 9lbs4oz and my third I ate really really well and was very active and she was 8lbs1oz... but my tiny one has continued to be tiny and my bigger one is now the same size as her older sister! Looking at my family and my DH's family the genetics are vastly varied! I had one doctor tell me that because my husband and I were both on the normal to tall side and average weight that our baby also needed to be that way. Dh has siblings that are very small though, a sister that is barely 5ft and a brother shorter than i am and both super skinny, so to me it is not odd to have a very petite child.