No I don't think it unreasonable, but I do think you should make a big effort to get your DD to see her grandparents. I know, I know it's their problem why should you have to do the extra work? But really to me family is so incredibly important and you are going to hurt your husband and your in laws feelings when you stay away from their home. Hoarding is like an addiction, and it is very difficult to break. Chances are they had this problem when your husband was growing up and so he is probably defensive of his family. He might even feel your rejection of his parents' home is a rejection of them and by extension, a rejection of him.
Help your inlaws understand that there are just too many pets for you child to be around, but that you want them(the inlaws) around your DD. Plan activities that they can come to, meet them at a park near their home, be proactive about getting time together and you won't have to worry so much about hurt feelings and resentment.
And I would add that I think it might be worthwhile for you and dh to have a counseling session together at which you can discuss the issue with a professional who can help you talk through it. I say that because it seems like for your dh there might be a lot of emotional baggage attached to the situation. Perhaps a counselor could help you guys navigate the conversation in such a way that you could avoid hurt feelings.