When my boys were 5, I knew all the parents of the kids who invited them to parties and we almost always had family parties. Occassionally they were drop off parties, and the parents usually had one or two helpers or grandparents there to assist. If you want it to be a family party, I think you need to just say so. But by 5, many kids are ready to be dropped off and can cooperate with the parents holding the party as long as you have someone else there to help you.
I do think parents have varying expectations regarding parties. For instance, my 6 yo was invited by a kid in his class (at the beginning of the year, so we didn't know him or any of the other kids) to a swimming party and the parents told me to drop him off, or that I could come and pay my own admission. (I paid my way.) My son is an excellent swimmer, but he still needs to be supervised. It turns out those parents didn't even get in the water with the kids and were just relying on the lifeguards. I really don't care for how thoughtless they were about the party and I surely won't accept another invitation from them. And then there's the invite my 3 yo dd just got-- they wanted me to drop her off because their house wasn't big enough for a lot of parents. Um... NO! I don't know them and don't really know the birthday girl except from preschool drop offs and pick ups. If their house isn't big enough for me, they shouldn't invite us!

Three is too young for a drop off unless its a relative, IMHO. I think by 5 it's OK depending on the circumstance. And a swimming party? No way. Maybe if they're 10 and it's the swim team!