Have you tried going for sole legal custody? At that stage, you would no longer need his signature.
In your situation that's what I would do. If you can prove he is not a good parent and can win in court, that'll sort you right out. Doesn't mean he can never see his son again, just that he can't completely ruin his life by not allowing you to go on holiday with him.
We just had to get passports for my DSD and DSS and luckily, heir mother was completely co-operative, but the next step would have been going to court and filing for sole legal custody - something we would have won, given that she has not seen her children by choice for two years.
I know there's aform, you see, where you can justify not getting the other parent's signature, and you might get away with it - but - at immigration, they may ask you for proof that the other parent knows you are leaving the country, and at that point you'd have to produce a letter signed by him, saying it's alright - or something of that nature.
So honestly, in the interests of your son and his future, I'd simply go for full legal custody. Again, that doesn't mean your son's father would be unable to see him - just that these kind of legal stupidnesses would no longer occur at all.
*HUGE hugs* XXX