I would say 4 and 5 are old enough, for most kids (probably not all) to be outside in their own yard, certainly. I think 5 or 6 is old enough to be riding bikes in the neighborhood, for short distances, with friends or to a friend's house.
IME, this is actually a bit conservative (but I am on the conservative side, too). In my neighborhood, which is very safe, and quiet, parents routinely let their children out unsupervised from 3 years old on. Not just in their yards, but up and down the street with older brothers and sisters. The 3.5 year old my ds is friends with rides a small 2 wheeler with training wheels. He is allowed to ride all up and down our street, ON the street! His parents are rarely outside at the same time. This blows my mind. His father laughed at me for not letting ds ride his trike on the street (when we do have sidewalks)--"this street is so safe, everyone is watching for the kids".

I think physical abilities come long before the development of good judgement. This little boy almost got hit by a car the other day by an older woman trying to back out of her driveway, but the little boy was oblivious. No one had taught him to wait for cars, watch if they have their lights on, etc. There are a couple of other times where we were out playing and found him upset and helped him get home. It is a community and I guess people feel like all the adults are looking out for the kids. But I think until a kid is elementary school aged, the person looking out needs to be the parent.