I did this last year for my 5 yo. I rented a pavillion at a Forest Preserve. I did a brunch at 10 am - bagels, gogurts, juice boxes, fruit and then we had blueberry muffins instead of a birthday cake.
There wasn't even any play equipment and I didn't have any games planned. The kids had a BLAST! They just ran around the field and played the whole time while the parents sat around the picnic tables and talked. Several parents reported to me that their kids said it was the best birthday party ever! I was really worried before hand that I didn't really have anything planned, but kids know how to make their own fun, literally in this case out of nothing.
I think you could totally do it without the pavillion and just set up on a grassy field with blankets for the kids and if parents are staying ask them to bring their lawn chairs. You could always bring some bubbles, frisbies, jump ropes, etc.
Hope you have a great time!