I know what you mean. I'm not gonna lie, our grocery budget has gone up quite a bit since eliminating a bunch of stuff from our diet. But looking back on it, I think we used to rely way too much on cheap wheat as the foundation of our diet. It's not the worst thing in the world for you, but it's not very nutrient-dense either. Going through this transition has changed my family's diet for the better. We are eating a wider variety of foods and a lot fewer processed foods.
There are some things you can do to keep costs down. Corn tortillas are really inexpensive and can stand in for wheat tortillas and sometimes for bread. You could do wraps for lunch instead of sandwiches. You can also use lettuce leaves for wraps. For breakfast, GF oats are more expensive than regular but are still way cheaper than storebought waffles. DD and I also often have fried onions, potatoes, and red peppers for breakfast. We've done a lot of dinners that have baked potatoes (or some other form of potatoes) where we would have had bread before (e.g. soup and baked potatoes or stew and sweet potato fries). You can also base dinners on rice, quinoa, or other GF grains instead of noodles. And if you make extra of whatever you make for dinner, that takes care of lunch for the next day too so you don't have to worry about sandwiches.
IMO it works best if you don't rely too much on directly subbing special GF products for wheaty products. It's fine to do that some of the time, but it gets way too expensive and honestly I don't think it's that good for you anyway. If you need meal ideas or other forms of support, you can check out the Allergies forum. There are lots and lots of GF mamas on there and even some who are veg*n.
ETA: I have found the best prices on GF products at our local natural foods store--they seem to be way more expensive at the regular supermarket. And Costco has some GF stuff, including Bob's Red Mill Gluten Free Flour blend.