These products might not be toxic in your garden or pots, but they are not benign! There are environmental impacts, and long term health issues for workers in the case of vermiculite. Reconsider using these extensively!
The issue here was asbestos contamination in the mines and products. I'd be interested in an update:
A fairly balanced article about peat moss extraction: