If you were never married (I assume so since he's not on the birth certificate) you have full physical and legal custody by default. He has no legal rights to the child unless he petitions the court to grant them. Doing so will trigger a child support order as well.
So if he doesn't like the amount of time you share with him and he wants more, he can pony up and get a lawyer to help him file for visitation rights ... though he'll be ordered to pay child support too.
If he's an absent dad and the issue is that you want him to pay child support, you can file for that. It does come with the risk that he'll counter-sue for visitation rights.