I wouldn't do it. Even if it's "not too hot" outside, the temperature inside the car can skyrocket very quickly. Several children DIE here in Texas every summer because of this very thing. (I read once the actual physiological changes that cause death from being locked in a hot car. Heartbreaking to think of a baby going through that. Don't leave your dog in there, either.)
The hot car thing completely terrifies me. I hate having my kids in a hot car even for a moment -- like if I put the baby in the passenger's side and then have to walk around to the driver's side to get in myself, it gives me the weebie jeebies. What if I had a heart attack on the way there, or I was mugged and injured? I know that's totally paranoid, but I do think it and hate having it even pass through my mind. And my car windows are tinted dark (which I hate!) so if somehow my little one were in the car and something had happened to me I don't think anyone would be able to see him. BUT this is not an issue in *all* weather. You only hear about that happening in Texas in the summer because it really does only happen when it's summer-y weather (and yes, that can happen any time of the year in certain locations -- it was in the 80s here last week). I understand that it can happen at lower temps than one might think, but it would not happen if it was say 55 degrees out and you were parked in the shade.
I was so relieved when my older ds *finally* reached the age where he could get out of his buckles on his own. Little guy is almost 3 but that main buckle is so hard to undo! But we are close I think.