You're not crazy, but you are lacking information.
I suggest getting Gill Rapley's book "Baby Led Weaning", or checking out websites with that title, for more information about why choking is *not* a huge concern.
In brief, it has to do with the gag reflex, which is a protective mechanism. When food (or anything else) gets too far back in the mouth, unexpectedly, then the gag reflex kicks in and you hork it out again before it gets to your airway. In adults, the gag reflex is located all the way at the back of your throat. In infants, it's much further forward, so it's activated much more easily. This makes completely logical sense -- when infants are in the stage where they are learning how to eat -- how to chew, how to manipulate food in their mouths -- it protects their airway while they figure things out. It then gradually moves back over the next few months to a year or so.
Spoon-feeding actually bypasses the gag reflex, so it loses its protective mechanism. Plus, infants lose the opportunity to learn how to control food in their mouths. Once they're allowed to start some foods that require chewing, it's often much later and the gag reflex has already started to recede, lessening its protective function.
The practical result of all this is that babies feeding themselves real food will often *gag* -- they'll make "bleahhh" and horking noises and stick their tongues out and food will be forced out, sometimes it even looks like they're puking... it looks and sounds terrible, but it's completely harmless and normal -- it means the protection is working. It's very different than *choking*, where they make no noise at all and nothing comes out. We often say someone "choked" on a bit of food when we really mean they "gagged", it's definitely unpleasant but not dangerous. "Choking" means your airway is completely blocked.
Besides... not offering real foods early on for fear of choking doesn't mean they won't choke. They can aspirate purees. And older kids, fully weaned onto 'real' regular foods, can still choke. My niece choked on an apple piece when she was 3 -- well past all the "don't give them chokable foods" caution ages. And yes, this was truly 'choking' -- turned blue, passed out, went to the hospital. (She was fine later) Having done everything the 'usual' way with purees and gradual transition to solids (and she was still bf at that time too) didn't mean she was any less likely to choke.
Oh, and just a note that the term "baby led weaning" refers to 'weaning' in the British sense -- weaning onto solid foods. In North American we tend to use weaning to mean "stopping breastfeeding" -- it's more accurate though to mean the *process* of transitioning from fully bf to fully solids, and that process can take months or years to complete. BLW is a concept completely about starting solid foods and has nothing to do with how long you choose to breasfteed, or even formula feed, although since the infant is in charge of the transition and not coerced into an artificial imposed schedule, they can be more likely to breastfeed for longer.
Anyway, I'm sorry I don't have much info on your bm/formula allergy dilemma, but I did want to reassure you about the choking thing and encourage you to look more into BLW. There is some evidence that it's especially valuable for kids with allergies, many parents report that their children will naturally avoid the foods that they're allergic to when we stop interfering!
Also, when dealing with severe allergies, you've already got enough to worry about, enough work to do -- as you already know in regards to your own cooking and meals. BLW is SO. MUCH. EASIER. than purees, it's one less thing for you to have to stress about.
Oh, and the book and websites have more info about common sense steps to reduce choking worries (shapes of food pieces, etc) -- I do suggest reading some before just diving into it blind, since you're a little nervous about it.