I think you need to be straight forward with your dad. When I had my first babe; I was constantly challenged by family on everything. My mom is a master of manipulating with guilt. It caused me a great deal of stress. Finally I could take no more of it. I told my mom that this was the way we were raising our child and that we would just have to "Agree to disagree."
How about something short and sweet? Such as "No dad; baby stays with me. Period."