Definitely listen to your instincts.
FWIW I have two close friends with babies who could still wear size 3-6 months at 1 year. Their babies were happy and healthy despite slow weight gain. They met or exceeded all milestones so the mothers trusted that everything was as it was meant to be. Now those kids are healthy 8, 7 and 4 year olds.
On the flip side, one of my friends whose son I'm donating milk to stopped growing due to allergies. He was born weighing 9 lbs 11 oz (so almost 10 lbs). At 3 months he was still hovering at around 11 lbs. She had a breast reduction and cannot make a full supply so she relies on donor milk and formula as supplementation. He started getting sick on even a tiny bit of formula. She started to suspect allergies since they have a family history. Once she got rid of all the donor milk that contained allergens, went on an elimination diet and got donor milk from me since I was also on an elimination diet, he started to gain rapidly. He's been gaining around 4 oz a week since 4 weeks ago. Other than slow weight gain, he seemed happy and healthy, but she just had a sense he wasn't gaining as he should. Her other kids gained much faster at the same age and they were all born similar weights.
ETA: I have another local good friend whose son was posteriorly tongue-tied and didn't gain much weight the first 3 months despite nursing all the time. She was finally encouraged to take him to an ENT who diagnosed the tongue tie and clipped it. He is now 7 months and doing remarkedly well. He started gaining weight right away and got chunky within a month. Before then, his mother had no idea anything was wrong. His latch never hurt so she didn't think it was a bad latch.
Di Linh, mama to DS1 (7), DD(4), DS2 (b 12/01/09)