Now my ds loves fire and fireworks, always has, but he would have been freaked out by the noise and surprise of firecrackers being lit nearby. And you are one smart cookie to not give little ones sparklers! They don't understand the reach of them (how many little kids misjudge sticks they are holding and bump someone?) Although I think those popit things are pretty harmless (I've snapped them between my fingers, there is no heat at all), in that context of surprise, sparklers, and firecrackers, I'd expect a young child to be leary of them.
Your 6 yo is probably about old enough to handle sparklers under supervision, assuming he has reasonable impulse control. But then you have the situation of a bunch of younger siblings that aren't ready for that. You don't want them all crowding around or let the big kids do it but not the little ones. That's awkward in a big family.
I don't like firecrackers. There is something about them (the small size, I think) that lends themselves to people lighting them while holding them and then blowing their fingers off if they don't drop them. I also don't like anything that can be a projectile like bottle rockets. They can fall over or get knocked down and go sideways into the crowd. People also like to hold them and light them, shooting them out of their hands. Then there are things that spin around on the ground shooting sparks. But if they hit a bump they can go into a crowd, too. I like the fountains, things shaped like canisters that stay in one place and shoot up sparks. And since I just have one child, I have always been able to supervise him very closely. I don't like doing fireworks when there are younger kids around.
So, coming from a mother of a fireworks loving kid, I think you are being very reasonable!