Febrile seizures occur when the fever rises rapidly to a high of 105 or 106 - not from the actually temp itself. It's the spiking of the fever to watch out for more than the actual temp.
In fact, when a fever is due to illness, the body will naturally prevent it from rising above 106 degrees. Only in instances of poisoning and heat stroke will it rise above that and of course then you do need to treat it immediately and seek medical assistance.
This information is from Dr. Mendelsohn, a renowned pediatrician who wrote, 'Raising a Healthy Child in Spite of Your Doctor".
In most cases it is better to let the fever do it's work, (fighting the illness) but as I've written above, there are instances when it is necessary to seek relief and/or medical attention for a fever.
My daughter's fever reached 105 a few weeks ago and it is indeed scary to watch the thermometor continue to climb! However, she was sleeping peacefully, had been hydrated throughout the day (wet diapers) and her temp had climbed steadily and gradually throughout the day rather than rapidly and suddenly. Therefore I let her sleep and did not try to bring the fever down. I let it do it's 'work'. The next day she still had a low-grade temp but was acting content - just a bit more subdued than usual.
If you must treat the fever, use coolish washcloths as suggested above. This may alleviate some of your son's discomfort.
And yes, you're the mom - trust your instincts!
Good luck - poor thing! I hope he's feeling better soon . . .