Ummmm... I think it's kind of offensive. I'm not saying that to start a fight. I'm really not. But if I saw your son in the playground, I'd say something to you about it. I can elaborate on why, if you want.
1) I'm not 100% cool about t shirts that divulge children's conditions. It seems there's a right-to-privacy thing there. Maybe that isn't a factor for you--some people don't believe that sort of right applies until a child is older. Still, I do think it's unfair to indiscriminately broadcast someone else's condition to the world at large.
2) He may not be processing the "dirty looks and disapproval" in a way that seems to you that he "understands" them, but if my experience and those of other adults with ASCs holds true, he is aware of what is going on and it will stay with him when he's older. Whether you are supportive of him and his differences or embarrassed about him will make a tremendous difference in how traumatic these experiences are--YOU are much more important to him than someone on the street.
3) The statement on the shirt would seem to imply that people with AS wouldn't "understand" the responses of others around them. Someone reading the shirt who had no knowledge of what AS is would probably assume your son had limited cognitive function, based on the "I don't understand you" message of the shirt. If he indeed has AS, that simply isn't true. (ETA: he might not understand because he's a toddler, but he's perfectly capable of understanding with AS.)
4) I do not like it that the shirt purports to speak in the child's voice, using "I" statements. Presumably, he has his own voice, and does not need to wear billboards proclaiming what mom wishes passerby to know about him.
5) The final statement on the shirt ("but my mom does") brings it back to being all about you. It really isn't--or, it shouldn't be--but if the point is that you are fed up with the looks/stares/attitude, then why force your son to wear the advocacy-gear? Why not buy/make/wear a shirt yourself? I've seen a number of moms with the "Proud of my autistic son/daughter" shirts. It's a tasteful way to get the message across.