We had also delayed the MMR shot for our son, as it was the one that concerned us the most (at the time). We delayed it at his 18 month appointment, and his 2 year, and 3 year, and 4 year appointments. After much more research, neither of our children will ever get that shot (as well as any other shot for that matter).
That voice in your head that is making you "freak out".....that's your mothering instinct. Listen to it. If you are uncomfortable with your son getting the shot - don't do it. I came to the conclusion that I had more qualms with the MMR shot itself than I did with any of the diseases that it claims to prevent.