I second the suggestion to keep records of everything. Every time he sees her, every time he calls, every time you call, every time you offer time to let him see her and he refuses, etc. I've been keeping records like this for 6 1/2 years. We've been to court twice and they've been very handy to have. I'm pretty sure we'll be back in court in the next few years and, again, it will be handy to have.
Your little one is right about the age my ds was when ex first took me to court. Up until that point I had been bringing ds to meet ex at least once a week, more if he asked. Most of the time he didn't show up. He got a bug up his butt and decided he was going to go for every other weekend, every wednesday night and I think he also requested 2 daytime visits every week. At the time he asked for that he had never been alone with ds. DS was 8 months old and still nursing.
I freaked

And then somehow I managed to calm down and in court my lawyer and I proposed a schedule for building up visitation instead of throwing ds right in. I suggested ex start at 1 hour visits 3 times a week, then move to 2 hour visits 3 times a week, then 3 hour visits 3 times a week, then 4 hour visits 3 times a week, then 5 hour visits 2 times a week and a 3 hour visit once a week..... I can't remember where it went from there but it eventually lead to overnights and weekends. The Judge was impressed and convinced that I really wasn't out to be a b*tch to ex, I really was keeping ds's best interest in mind. He ended up ordering visitation at my discretion.
Ex worked up to 3 hour visits then refused to move to 4 hour visits. He rarely showed up for visits as it were and then when ds was 25 months old he just stopped showing up and calling. He's not had unsupervised visits since then (mainly because he's disappeared from ds's life and I refuse to make ds go with someone who doesn't have a clue about him- ds has autism). Ex has gone up to a year without any contact (no calls, emails, texts, doesn't see ds, etc). Yesterday marked 3 months since we've heard from him this latest round. The last time I heard from him was an email (after he skipped a visit with ds without any calls or anything) saying that he believes visitation needs to change so will be taking me to court to attempt to get all school vacations (christmas, spring) and half the summer. I sent him a brief email reply stating that it was his right if he felt the need to and made sure he had my address to have his lawyer send paperwork to

Haven't heard from him (or his imaginary lawyer) since.
All that to say.... I would recommend you find out the laws in your state (regarding how old a child has to be when overnights are ordered) and then make a schedule to build up to overnight visits, since it sounds like there isn't any reason a judge wouldn't eventually order that.