Yes you should be worried about it. Rock eating can cause intestinal obstruction, which is life threatening and needs expensive and risky surgery to fix. They can also break teeth on rocks. Someone in my family just had TWO of her dogs undergo surgery to remove rocks from their bellies this week. Apparently they decided to re-landscape the yard to remove the rocks after this incident. Is there a way to keep him away from the rocks?
I'd probably keep him away from the rocks and/or leashed and orally busy with appropriate toys. If he does get a rock, carry something to trade him for it with. Right now, if you take the rock away, all the fun is gone. You might have to carry a treat pouch and bribe him with the treat to drop the rock, then offer an appropriate chew toy. (You do the same for anything else he should not chew- like shoes, chairs, etc)
Another thought might be to find a good size rock (large enough to NOT swallow) and spray it generously with Fooey or Bitter Apple, then set it down where its liable to get chewed. Those products taste awful to most dogs, and they'll get a mouthful or two and then decide that object is not worth chewing. You can use it on furniture and other things you don't want chewed. Just remember to reapply frequently at first- like once or twice a day.
The best thing would be to start working on the "leave it" command. There are numerous ways to teach this. I bet stardogs has some good ideas. The idea is, the dog learns that if you say "leave it" they can not touch the forbidden item. It is REALLY handy. I'd almost say vital. You would not start training with rocks, but the idea is that if he so much as looks funny at a rock, you issue the command and he ignores the rocks. I've found it to be vital for safety situations too- for instance, perhaps you drop dinner on the floor, or some pills (and don't we all at some point), an untrained dog would wolf it down, but leave it gives you time to get the dog contained and the dropped items picked up with noone getting hurt.