In Washington state (the only place I have any knowledge of) a child must be 5 by August 31st to start kindergarten so 6 by that same date to start 1st grade. From what I have seen, they are pretty strict to it. If you want to "test in" with a child who misses the cutoff by just a little (I think no more than a month will they even consider), there is a six part test and the child must pass all six parts.
I have a youngest in class dd (summer birthday). Even though she was very aclimated to the school idea (did wonderfully in two years of preschool) and was advanced academically, kindergarten was a little challenging emotionally - she got tired and frustrated easily. No one was more surprised than me! This year 1st grade is such a breeze. I would not sign up for youngest in class if her birthday had not fallen that way.
Grammy was a preschool teacher for 20 years and thinks they should change the cutoff to June 1st so you don't get those "just turned five" kids. It can be hard for them. It is not an academic thing at all - more a maturity level which someone else mentioned is hard to determine until they are in that exact situation.
I would suggest looking into multi-age schools in your district. They are free programs of the public schools. I had not heard of ours even though I lived in this town for 10 years before dd was old enough for kindergarten. That way, he can "be" in kindergarten but work ahead if that is where he is in whatever area.
Also, you mentioned he is on the small side. Another reason to go ahead and let him be a kindergartner. If you get a good program, they will challenge him. Being the oldest in a class is never a detriment IMO.
Good luck with your decision and congrats on having such a bright kid!
A lot of us MDCers seem to have them too!