Depends where you live. I am a big "public spaces" booster, so if there are parks/playgrounds w/in walking distance, I probably wouldn't - I love encouraging the kids (and myself and the sitter) to engage in the community, meet other kids, etc.
Or, if you have neighbors who have sets and your kids are truly welcome to play, I might not. My kids are at my parents all the time, and they play on the neighbors sets, with the other kids (always supervised by us though, and of course we're very respectful of their things, space, etc).
I live in an apt. We have a little playground at our complex which is great - it's like a backyard, but used by about 7 kids, so there's almost always a playmate!
If none of the above apply, it would probably be really fun for your kids
