After reading your post, I had to think about what I would do in your shoes.
I think you're right about it being something that he'll pass through. Since he's ok with wearing clothes when you go out in public, I wouldn't worry about it around the house.
As for your glass door, it sounds like you don't want him flashing people because of his safety. While you're concerned for his safety I want to point out that it is more common for children to be molested by people they know than by a stranger. Does your glass door open onto a busy area? Is it common for people to be passing by?
The thing is that you need to follow your intuition... if you're uncomfortable, then give him some simple boundaries with his body. I've told my son that his penis is special for him, no one else. Only he can touch his penis and when we change his diaper. (I suppose we'll have to also add the doctor at the doctor's office). This has only come up a few times when he has had questions about his body, for instance at bath time. At any rate, I want my son to feel comfortable with himself, his body, and as he matures his sexuality... Yet I also want there to be clear boundaries for safety.
Take this for what it's worth. While I'm not a nudist, we do have an open house in terms of being nude around each other at bath times and getting dressed. We actually don't have naked days, but I know some families who do.
None the less, you're going to have to do what YOU"RE comfortable with... And it sounds like you're ok with him being nude in the house, and not comfortable with your son flashing the neighbors. Are you comfortable setting a boundary about the sliding glass door? Like if you want to stand in the doorway you need to have your underwear on?