My older DD is on the spectrum, and my younger DD is gifted. For us, too, figuring out what was going on with our younger DD always took a bit of a back to seat to the much more complex issues of raising a child with autism. We felt relieved that she was developing typically, and left it at that for a long time. We didn't do anything specifically because she was gifted until she was 10.
The way it played out for is that for the first several years that they kids were school aged, we homeschooled in a relaxed way. Not because of special needs or giftedness, but just because that was my preference as a mother. For a variety of reasons, that stopped working for us and the kids started school at 10 and 12.
Starting school was a bit a reality check for our then 10 year old. Before, she only knew how she compared to her sister, but now she could see how she compared to her age mates. The first year she had some skills she needed to catch up on because they had never been important to her, but she did quickly catch up. By the end of the year, she had been recommended for testing in the gifted program. It was the first time we considered that she was gifted. I think it would have been more obvious to us if she had always attended school. Her basic personality is very mellow about many academic things -- she's more of a doer than studier. But being the best at whatever she's doing compared to those around her is very important.
She continued to attend public school the next year with gifted pullouts and enrichment and all that, and there were some social issues with that. She was called nerd a lot, but once told me that nerds are what parents want -- kids who do their work and go to school to learn. I don't think these issues were caused by the gifted program per say. She really enjoyed her experiences in the gifted program.
The following year we switched her to a private school that allows kids a lot of academic freedom and doesn't award grades. She's found her true peers there -- some very funky kids who bright and and challenge her. It's a place where the ability to keep your classmates on their toes is seen as a good thing.
One thing I recommend is not going exclusively on your own experiences but looking at the options your child has. The gifted program at his school isn't the same gifted program you were a part of. Your city may of other options, such as charter schools or private schools that better match his needs. I think that having true peers is good for any child, whatever their IQ or challenges.