On my own with dysfunctional alcoholics and other behavioral problems within my own family. With DH in issues associated with his family of origin. My DH is a strong man but his father and Step mother could reduce him to a puddle. He could not stand up to him, it was like he reverted to that scared, abused 5 yo as soon as FIL started in on the manipulative, bullying behaviors. He made the decision to break all contact with them long before we had a child. They were a big part of the reason we waited so long to TTC, we were afraid that we could never 'get out' once a child came into the family.
You mentioned that your DH "said he didn't recall anything from his childhood that was unordinary like that" I say the following gently - your DH memory may be fooling him. In addition to my DH, I have several friends who are children of abusive active alcoholics/addicts as well as abusive "dry drunks" and we all have blocked out the really bad stuff at one time or another, it is a protection mechanism.
Doing some research on co-dependency might give you more insight into your DH and how he is (not) handling the situation. One of my big mistakes in our journey towards breaking ties with the ILs was that I could not keep my mouth shut about how I thought DH wasn't doing enough to improve his situation. My family issues were bad but no where near DH's. I didn't have the twisted, manipulative element going on. At the time, I had no idea where DH was in his progression towards independence, I was impatient, I didn't understand how hard it was for him, I was constantly on him to stand up for himself, to just "do something". Later, I learned how my attitude hurt DH. We often compare his relationship with the ILs to a woman who can't leave an abusive spouse. If you haven't lived it, you don't know how strong those emotional chains can be.
Who initiates the visits? FIL? or your DH? Have they always stayed in touch or is this something new?
There are some members over in Parents as Partners that have experience in similar situations, might worth popping over there for some advice on dealing with the DH element of this situation.