I was just wondering this myself. We don't have a pantry though, so I'm not sure where I"d keep it if I didn't keep it in my fridge (unless in the garage maybe?) I do find my tomatoes to get mealy though. And we don't eat a lot of potatoes.
Just wondering though, does it bring bugs around to have stuff out? I think I got spooked after this summer when we had a major fruit fly invasion because of bananas left on my counter. It was HORRIBLE. I had to start putting the bananas in the fridge and it made them awful, so we stopped eating bananas. That was only in the summer though. And our house is very clean, so that wasn't the problem. anyway, that's what makes me scared of leaving stuff out. But we do live in the south, so the summers get brutally hot.