Just some thoughts, I've been through this as I had court ordered extended visits for my barely 2yo. First, I made her dad work up to it in town...both by building a relationship over time and having one overnight, then two in a row, etc. She ended up handling it well, and having had his cooperation made all the difference in the world in terms of my comfort. Also, in our situation, she had her big sister with her, so there was that feeling of "home" all the time. I did have to send dd1 off on a 3 or 4 day trip with her dad before that...and she did fine. She was just turning 3 at the time. BUT she never hesitated or resisted going to her dad's, it was always a happy exciting thing, so that makes a HUGE difference to me. And that time, I made sure I was close by (stayed with family about 5 hours from where she was, rather than being a two day drive away), so if she did get overwhelmed, she could come home.
And just for the record, I think with little kids there IS a big difference between 3 and 4 overnights. Plus, if I understand things correctly, he'd be driving two days, staying a night or two, then driving two days back???? That doesn't sound like a good vacation to me, unless your child is really good at/used to being in the car for long periods, it could be pure misery on everyone's part, and driving is a hard time to parent solo.
Since you have discretion at this point, he's only had one overnight in town, your son shows reluctance, and the trip isn't going to be time together so much as stressful travel time, I'd say no. And I do agree that precedents can be set by things like this...if you agree to three or four days now, you've lost the chance to work up to it. If it was for something really special (like a wedding...that's what my dd went to, she was the flower girl) then maybe it makes sense, but otherwise, he will just have to develop a relationship with your ds and wait until ds is ready.