I'm wondering if it isn't a good plan to look further down the road. If you have shared custody there are just going to be times in the future where you x is going to be in your house (if you maintain the civil custody sharing arrangement you have now). At some point your older DC is going to need some help carrying something and your x is going to come help her, right? So, I guess I'm feeling like you are going to have to either address the issue of him not ever coming into your home...or you may have to just get comfortable with the idea.
Unless, of course, you're talking about worrying about him unlawfully entering your home when he has no cause to be there... Is that your concern?