We are ones who rarely use "babysitters" and the only ones who were ever chosen were grandparents or our best friend and is 19y/o son. My son will be 6 in Novemeber.
That said...I know that when he is with any of them he will likely get candy/junkfood etc...BUT...with 99% of his time with us you know what? In most cases he will choose healthy foods. He refuses to eat non-natural peanut butter-he hates it, and wonderbread sucks...he doesn't like dips and things for his veggies he just likes them raw and plain. He will complain to my mom that it isn't like how I make things. Sure he loves candy....but candy isn't off limits (we generally save it for Saturday and he knows this) and chips etc.
The way I look at it is that the occassional additional piece of non-healthy food won't be detrimental to him...plus he gets lots of grandparent time. I loved my time with my grandparents and looking back am sad that I didn't spend more time with them when I could. Sure I remember being able to have lots of penny candies and ice cream and everything else I couldn't usually have but you know what my favourite memories are? Learning to peel potatoes and apples with a paring knife, picking raspberries for her to make pie, learning how to make butter tarts, learning lots of ways to make potatoes as my grandpa had to have potatoes with every meal, playing games and doing puzzles with her, learning the basics of knitting and embroidery. Everytime we have raisin bread I think of my grandparents, when we have KFC I think of the big (huge) family gatherings we would have in the summer with KFC being the easy choice. When I have welsh rarebit I think of my other grandmother same thing goes with homemade cocoa. Scotch mints make me think of my one grandpa who always, always had a bag of them in the house. Homemade coconut cream pie is my favourite pie...my grandma made the best.
I would never sacrifice my son's memories and learning opportunites just because he might spend a day eating garbage food and doing things we might choose not to do with him. Unless it is endangering (I installed his car seat) him then I let them have fun and build memories together.