Perlite is just rock popcorn - puffed lava rock or something if I vaguely remember right. Vermiculite is the one with all the health issues.
Well, if everything in my pots tanks I'll blame it on the perlite.

I think it
is in soil mixes like Black Gold Seedling Mix though...
You could always see if you could grab some dirt from a neighbor or local friend or whatnot to mix in. Just peat moss would make
me wary, unless you're only planting pine trees or blueberries (or lilacs). You'd have to test the acidity of the soil to see whether you'd be comfortable with just peat moss. Or there's something like cocopeat - shredded coconut husk fibers. That might not mess around with pH as much, but could also be a little pricey.
I'm not as fussy as others about the whole organic label. If I know how/where things are grown and such, having the official USDA label is very much not an issue for me. If I can feel comfortable sending my kids out to my yard or whatever u-pick operation we're at and not worry about them snacking on whatever, I call it good. But it does mean I still don't want to use styrofoam peanuts in my growing containers.
