I'm horrified on your behalf! Though I can understand why you are reluctant to get the children screamed at you can still step in yourself. You are an adult and as such there for the guidance of children, especially when they are in danger! I can't see why you can't quietly point out to these kids when their behaviour isn't ok, perhaps step in if things get out of hand and assign them 'tasks' or a new game such as building castles etc. Children who aren't getting a lot of attention will often respond really well when someone takes an interest. Though the other parents don't do things your way you may be able to institute some positive change. Don't feel that you have to turn away from things because they are not your kids and don't feel you can't set boundaries. Explain to the little one's that now is a private time and that they should find something else to do. Also think about getting some blinds for your windows and don't feel bad about shutting them!
Good luck.
Good luck.