He's not going to get every Sunday. Even if you are off on Fridays, the kids are in school. It would be very difficult for you to plan week-end trips if he has every Sunday. And it is unlikely that a judge will order it. Be firm with your no. And don't even do it on a temporary basis as that will set a status quo.
If he moves back to the same neighborhood, chances are good that he will get more time. But you can have it set up to where you get them if you are off from work and he would just pick them up after work.
It really sounds like he is not going to be co-operative with you at all and will not follow the court order when everything is done. You need to discuss with your attorney what the consequences would be if the court order isn't followed, such as putting in there that should a contempt motion need to be filed to enforce the order that the person in contempt pays all legal fees of both parties. That may deter him from being too much of a tool.
You also need to address what happens when the kids are sick. Because if it isn't addressed and you keep the kids with you on his week-end, you would be in contempt.
Here are some other items to go over with your attorney:
1. How to handle switches-the way I would handle this is that both parties have to agree to the switch in writing, complete with signatures.
2. How make-up time will be handled should the kids not be able to go to their Dad's due to inclement weather or illness.
3. ROFR-what this is that if either of you are unable to care for the kids on your parenting time, you first have to offer the other parent the time. If they aren't available, then each of you are responsible for making alternate arrangements. Given his attitude, I would put in there that should he not comply with this, you get make-up time on his next week-end.
4. Who has final say on education and medical issues. Save yourself some court time as it doesn't appear that he is going to be co-operative and just may make a simple doctor's appointment a battle. If you have final say, after his input, he would just be spinning his wheels and if he takes you to court over it, he would be showing the courts what a tool he is.
5. How birthdays, holidays and school breaks are to be handled. Mother's Day is yours regardless of whose week-end it is, same goes for Father's Day for him.
6. Summer vacation: each of you get 2 weeks uninterrupted time. Each of you have to give your weeks by April 1 (or a different date). On even years, you get first pick, on odd years he gets first pick or vice versa. What this means is that on even years, he has to plan his uninterupted time around your 2 weeks, same goes for you on odd years or vice versa. I would also stagger the dates, such as when you go first, you pick by April 1 and he has to pick by April 15 and vice versa. Failure to pick by the dates means that the uninterupted time is forfieted and it will go to the normal summer schedule.
7. Be willing to do the week on week off during the summer as long as you get the kids during the day if you are available. It will show the judge that you are willing to work with him. And your kids are old enough that a judge would order this for the summer anyway.