At DS 4th birthday, he mostly invited preschool friends (and a couple of cousins). I think there were about 12 kids total, and with the exception of the cousins/aunt/uncle, only one parent and the invited child showed up. In my area, drop-off parties didn't start until 6 y.o. birthdays, and it is unspoken that the party invite is for the invited child/friend of birthday child and one parent to supervise them at that age.
For DD's 4th birthday, she wasn't all that interested in inviting school friends so we kept it to relatives & family friends, but I'm sure she will be for her 5th.
If you think you're going to get a large crowd regardless of of you word the invites, maybe schedule the party after lunchtime so you can just serve some easy snacks & birthday cake?