My dad mentioned to me that people think DS is a girl at the park sometimes because of his amber necklace and GN clothing. He was sharing an observation, not making a judgement (thank goodness). My response was, "Eh, people can think what they want. It doesn't really matter to me if others think he's a boy or a girl. He seems happy enough with the way he looks."
"You dress him like a girl." - Page 2
I have pictures from when I was about that age dressed in very obviously "boy" clothes and I turned out fine. I did ask my mom one time why she put me in boy outfits and she said she just put me in what she thought was cute. Good answer, I think! I WAS cute in them!
I dress my DD in boy clothes a lot, because they fit her body type better (long and lanky and the girl's clothes are cut short and show too much IMO). I also keep her hair short for practical reasons. Oh, and her name is "Charlie". No one even bats an eye.
I don't think the hat is "girly" at all. Even if it was, your son will not have life-long damage from it. That's just silly. Just learn to ignore people. My DS always wore boy clothes, but when he got to be about age 3 or 4, he decided he LOVED girl things. I just let him play with what he want. We did get some comments, but oh well. He's 7 now and still loves Hello Kitty and will wear girl pajamas, even when he has friends over. If I let people get to me, I'd have gone crazy by now!