It sounds as though you're doing a great job of sleuthing out your son's challenges and meeting his needs. I understand your desire to see the least suffering possible. Since after starting solid foods you will still need to be watching your own diet, it's simplest to prolong this event for a while, if it seems OK with baby. Sometimes early food interest is mostly teething and exploring. This provides a nice window for starting solids when parents feel ready for them, but a breastmilk popsicle or a teething ring might just satisfy this stage for now.
You will need to know that sometimes a baby will react to foods that were OK when in mom's diet/milk. Your body has a way of partially neutralizing proteins for milk. This said, yes, you will want to start with only the foods that were tolerated from your milk, but still avoid those that are known to be more common allergens, such as wheat or tomatoes. Avocados and bananas sound great. You asked about rice. I've never considered grains as great first foods. Their nutrition is so much lower than the milk they're replacing and their allergenic potential is high. If he's been fine with rice in your diet though, a little brown rice for him is fine, when the time comes. Squash, yam, beets, and strained asparagus are some other great options.
If you're OK with meats, they do tend to be the least allergenic and provide excellent "first" nutrition. Even for a family that prefers to be vegetarian, I think a little meat can be an excellent option for a little one with food allergy challenges. Organic turkey is great, and the homemade broth from it is just as important as, or maybe even more important than the meat. (There are no such broths sold anywhere. Store broths are mostly salt water and usually some form of MSG.)
The most allergenic child needs foods that mom seldom had during late pregnancy or breastfeeding. For meats, buffalo or lamb usually work here. For fruits and veggies, a trip to an Asian market or Whole Foods may produce some nice options, such as lychee fruit, persimmon, star fruit, daikon, and bok choy. Hemp milk works for many, as a mixing ingredient or occasional drink.