i'm in a similar situation- i am in NZ & am not allowed to leave the country, or even live outside this specific region, with my kids. i grew up in the US & have only been in NZ for about 4 years now. as nice as it would be to go live with my family now that my relationship has fallen to bits, their dad is very keen to be an active parent, so i don't have that ability.
so i understand your frustration, it's hard to be anchored to a specific area when you'd rather be somewhere else. luckily i like it here & was planning on staying for now, but just the idea that i *can't* leave even if i wanted to is hard! (well, *i* could leave, but not with my children).
the same happened to a woman i met here who wanted to go back to columbia, where she grew up, with her child, who is a kiwi. the judge said definitely no. judge's are very reluctant to allow children to leave the country when the parents have split up. if i were you, if you ever want to be able to visit your family, i wouldn't mention wishing you could live there- they might not allow you to have a passport for your child (or insist your X keeps it).