Pshh. That means nothing. If this judge had a problem with that he would have had a problem with that when this case first happened. Did the judge know you were moving in with your dp? Did your ex move in with his wife before they were married?
And, regardless of what he thinks, its a HUGE abuse of Discretion for him to use that against you - since its NOT in O's best interest to have extended visitation with someone he doesn't even know! Especially since its your ex's fault he doesn't know his own father!
I wouldn't worry about that. You're lawyer has to tell you b/c she just does. BUT - if that changes things I certainly hope you file an appeal to a higher court.
Besides, if your ex is concerned about this now, he should have objected to it like 3 years ago right? Seriously - if this causes you problems (the judge may yell at you about it, just listen and ignore it) it will shock the heck out of me.