It's just a long time to sit for a one-year-old, that's all! Perhaps make it more age-appropriate. He could maybe sit with you for a few minutes and have some bites of food, some himself, some assisted by you. Then, when he starts to get fidgety, you can let him down and he can play on the floor with some toys while you and your husband finish your dinners. Then, perhaps your son might like to sit on your lap and have a few more bites, then play again.
Gradually, his tolerance for sitting at the table and following your conversation will increase.