You are certainly not being unreasonable, IMO. The liability issue, property damage, and disturbance of your peace are all valid concerns. I can't imagine why these parents would *want* their kid to play in a stranger's yard unsupervised. From their POV, how do they know you're not a creep? What if you had dangerous stuff in your yard? What if you go to back your car out and the kid is playing back there? You shouldn't have to be paranoid about kids behind your car every time you leave.
Our neighbors kids won't even come into our yard to fetch an errant ball. They always knock on our door to get it back. Our backyard is fenced in, and the guy in the other half does have (small, nonthreatening) dogs in the yard, but that may be serving as a kid deterrent. In our old house, we did have a problem with kids being on our porch, in our yard, etc; but it was another duplex and the people next door didn't mind, so there wasn't much I could say about it. I did get rather upset when the little girl kept insisting that I let her in my house to play with my cats! I didn't even know this kid's name, which house she lived in, or who her parents were. She was frequently free-ranging the neighborhood, and I was pretty close to calling the police. She finally left me alone when I shut the doors and curtains and refused to answer her, and we moved out of there not too long afterwards. It wasn't a dangerous neighborhood by any means, but yeesh.
I like to think that, if I still live somewhere where there are neighbors, I'll have the good sense to keep an eye on my kid until it's old enough to understand respecting other people's space, among other things. I dunno, I guess if I was in your neighbors' situation and had lost my sense of decency and respect, I'd want someone to bash me with the cluebat.