I've noticed that women with longer torsos tend to carry "smaller" than women with shorter torsos...I guess because there is more room for the baby amidst internal organs? I'm short AND short-waisted, and in general I look bigger than most people I meet with similar due dates, even though it's my first baby.
One of my friends actually ended up with a scheduled c-section because they said her baby was under four pounds and could not withstand delivery. She is very petite (wears children's-size underwear) and carried small. But when the baby was born, he was over six pounds! I think they just assumed the baby was on the small end of the ultrasound measurement (which is always +/- two pounds) because she hardly showed. I chalk it up to her torso...even though she's petite, her torso is really long and that baby had tons of room to grow without pushing out her stomach!