We have some of the same issues here. My daughter is 4 and my husband has always expected things of her as if she were far older than she actually is. Fortunately, she is oddly mature for her age, so it has worked out fine.
My husband does not like children. This isn't to say that he dislikes them, he just isn't interested. Children weren't ever something he "wanted" just something in his mind that "just happened," like it's part of life, like going to college, getting married, buying a car, etc. Like it wasn't a blessing to hope for, just another life event that happens in a certain order on your way through life. I, on the other hand, obsessively wanted children and hounded my husband about it for years until he finally caved and we actually started trying. I get a lot of negativity from friends for that because he didn't particularly want kids, but I see it differently. He didn't want kids, but he loved me so much that he wanted ME to have them. I see my children as a gift from him almost and I love him every day for it.
Another way he is different is actually sort of part of the above. He's terrified of babies and has never wanted to hold ours. They cry. He doesn't understand the crying and he doesn't know how to make it stop. It really freaks him out and if they make so much as a happy coo, he hurriedly hands the baby back to me. He would prefer I put the baby down somewhere if I have to do something rather than have him hold the baby. When *I* hold the baby, I can see pride in his face. I can see that he is proud of his wife as she holds and nurses his children and he shows off pictures of us to people he knows, he just won't touch babies. Once they get older and are completely verbal and less breakable, he's better with them.
Like you, I have had to be the #1 parent. My husband doesn't have empathy so he can't deal with things like crying ("stop that right now!") or discipline. He was spanked, so all he knows is to spank which, of course, I won't allow him to do. Now that we know he has AS, I've been able to help him learn other methods of dealing with emotional issues, but he still has a lot of trouble and we both agree that the best thing for him to do instead of losing it is to just hand discipline duties over to me.
That puts me in the position of top caregiver and with NO family here and no one to help me, I have not only been the head caregiver, but the ONLY caregiver. My children are always with me. I don't get "me" time like other moms. I'm always the one with them. I have been away from my 4yo three times: once my husband took her to a bouncy castle park while I went to a 2 hour midwives symposium, once when I had to go to the ophthalmologist and he took her to the park, and once when his parents flew out to visit and took her to the zoo for a couple of hours. It was totally bizarre to have alone time.
But, like I said, he's a great dad in his own way. You can see he loves kids kids and he really tries to make them happy. He plays with our daughter, but he's not good at pretend. They like to put things together. They build these incredibly intricate inverted pyramids with blocks (I have NO idea how DH does that!) and he bought her a marble run that they play with. He's the type to get her construction sets and then sit and build some amazing thing with it. He also will bring home surprises for her from time to time. It's not the "buying their love" type of present, just the type that lets her know that he was thinking of her - like flowers on her birthday this week on his way home from work. Sometimes it's something totally random but fun like glow sticks or silly string. He will spend time with her because he knows that quality time is valuable to kids, even though he doesn't really want to spend time with her. You can see how uncomfortable he is. It's the fact that he does it anyway, completely of his own volition, that shows how much he cares about doing what's best for her and what will make his family happy.
He loves us, just can't show it in a lot of ways, but he does have his own ways in which I can see he's making an effort.