I have had a few plants in the garden that the ants have gobbled up. I know this because I watched them do it.
I figure they were here first, so they have a few rights on my street!
Now, dh would like to exterminate all insects within the province. I have found rouge ant traps tucked behind rocks in my garden (did you know a special fairy brings these? I was not aware of it!) and have gotten into the habit of checking his pockets for violent bug killing chemicals when he leaves the plant shops.
My ants are under snow for a good chunk of the year (alot of snow!!) so they may be different from your ants!! But aside from the occasional REALLY tasty flower, they don't do any harm here. They seem to have no effect on the earthworms.
I really don't have much to offer beyond what Chanley said. They ARE doing their part in your garden. And if you get the odd, bored moment, they are FASCINATING if you get right down and watch them. I want to keep them around for the entertainment value!!! (says alot for my comedian hubby, eh?