Another poster already wrote this, but I thought I'd mention it because the healthy alternatives that have been mentioned (with a few exceptions) have similar risks. A few examples of alternatives that still cause dust and are known carcinogens to those with a lot of exposure are any of the food items mentioned (perhaps except for flax seed) such as flour, rice, wheat, corn meal etc.... Bakers are at risk of developing emphasema and other respiratory disorders as a result of being around it and working with it regularly.
Silica is a naturally occurring substance that is derived of quartz stone, which is pretty hard to avoid, anywhere!
I do think the pea gravel is the best alternative simply because it doesn't cause as much dust, but just don't think that you're avoiding the risk by substituting corn or wheat for quartz or even by using pea gravel unless you know of its origin. Stone contains so many elements in so various concentrations that you would have to test the very sand you were considering before being sure what contaminants you are dealing with. For instance, in south-central Ontario, there are areas where much of the stone naturally contains uranium that when left alone does nothing, but when the stone is broken (even just into pea gravel, and then especially sand) is radioactive and extrememly hazardous.
We're building a sand box this summer and will probably put pea gravel into it. It contains quartz, and so many other types of stone/crystal that it would take us the whole summer to find out what they are! Actually, that sounds like a great activity! Thankfully, there is no known uranium deposits up here, so at least that's not a concern. I think quartz is really one of the least concerning because it is a very stable crystal overall; it's rated at 7 for hardness and the hardest rating is for diamond at 10. Only Topaz, corundum and diamond are harder than quartz, so if what you have to work with is quartz, and you can't afford to fill your sand box with diamonds, I think you're doing okay.

Have fun! A pocket guide to rocks and minerals may help to dispell (or create) concerns. Either way, it's great to be informed!
Follow Mothering