My dd was like that. She started out average, but by the end of the first year she was near the bottom of the growth charts. Soon after that she was WAY below average. At nearly 5 she still only weighs 31lbs (as does her 2 year old sister).
I am not worried about her. I occasionally have my moments (I am her mom, it is my job
). But she is full of energy, she LOOKS healthy, her hair and nails grow like crazy (so she is obviously getting lots of nutrients, those things often slow down or stop if a child is failing to thrive), her eyes are sparkly and full of life, and she is otherwise well. She does grow, just very slowly. And I am not large or tall, so she likely just takes after me.
OTOH I DID have a child who suffered from malnutrition. He was a very limited eater and needed feeding therapy to learn to eat different textures. He was still at the midpoint of the growth charts and was a large kid so it took almost a YEAR of me fighting doctors to take me seriously. It wasn't until I refused to leave without a blood test that I finally got help for him.
The difference is a big one. He was pale and lethargic. He had NO energy. He was clearly not thriving at all. Growth charts and averages mean nothing. Judge by the behaviour of your child. It never hurts to be cautious, but IME the difference between a thriving child and a struggling challenge is a big one.
YOU are the one who knows him best. Maybe just bulk up his diet with healthy fats if you are concerned about it. Otherwise you probably can tell if he is thriving or not.