I think you may be in a position where you need to start figuring out if he's just 'picky' or if he actually has food aversions/difficulty with food.
If he eats only eggs and dairy and a range of whole grains and fruits, I honestly wouldn't be worried. He's getting protein from nursing, from eggs and dairy. He's getting vitamins from the fruit and grains. I know very few kids who will take more than a bite of brocolli! My kids have two veggies they will eat consistently: peas (fresh and cooked) and fresh carrots (and heaven forbid they should be mixed!!). Ds will eat tomatoes and fresh beans in season, but not much else. Neither of my kids will eat legumes because they don't like the feel of them in their mouths.
Which brings me to: Another thing to look at is texture of food and whether this could be a sensory issue. If it is, it's something that might benefit from occupational therapy in order to help him deal with the sensory aspect of it.
So, take a look at what he will eat - what does it have in common, taste-wise and texture-wise. Some kids will eat only smooth things, some crunchy. Some will eat either smooth or crunchy, but nothing in between. My dd is like this - she took a long time to get used to certain foods, night nursed until close to 3, and is very cautious of new textures. I suspect she's also a 'supertaster' as she's VERY sensitive to spicy foods, and has incredibly sensitive smell (especially for a 3 year old).
But she doesn't strike me as ever being as extreme as your ds is. We've had real success with introducing things slowly, kind of like a pp said. Put it on her plate, let her play with it. Then smell it. Then take a bite. Many things, she's said 'no' to. But the other day, she discovered she liked watermelon!
Somewhere I read that it takes 10 or more introductions of a food before a child will often try it, so don't give up.
Other books I would suggest:Just Take a Bite: Easy Effective Answers to Food Aversions and Challenges
by Ellyn Satter. She's got another book called "How to Get Your Child to Eat: But not too much". I've never read it, so I don't know how good it is.
If you think his issues might be sensory, then I recommend:
The Out of Sync Child
I would also keep a food log to see just what he DOES eat and see if you can work on expanding it from there. If he comes home from his dad's hungry, it could be for a variety of reasons. My dd is always desperate to nurse when she's been away from me for a while. It has zip to do with hunger.