I kind of think he might have a point. It's got a HIGHER skin to volume ratio than a bigger potato (because volume increases faster than surface as you size up), plus if something does breach the skin the potato as a whole will go bad quicker, instead of just being a spot you could cut out of a larger potato.
On the other hand, he and I could both very well be misled by the fact that a smaller potato will COOK faster than a big one, and we're making a false analogy to other processes! And as a confounding factor, tender little new potatoes are tiny and also don't keep well, compared to a large baking potato -- but that's not because they're small.