I just want to give you big (hugs).
I've worked with a lot of families where there was incest and what your sd did would qualify as that. Check out www.rainn.org
for resources and to locate counseling.
Of course this would be hard to deal with. It's a nightmare. (gone thru it with my sister and a relative of ours)
In families where this occurs there is a lie that must be maintained. That the touching is not happening and that everything is ok and normal. To keep it's balance the perpetrator and their partners will pull all kinds of tricks to make the victim feel like they are the crazy one or the bad one.
He is the one responsible here.
Get support. Tell someone. Get Counseling and keep him far, far from your precious dd.
Pedophiles-by nature are proflific abusers who have an irresisitible urge and often really do believe they love children. They may be attracted to one specific age and gender (or not) They are often in relationships with another consenting adult. But anyhow, if he is a "pedophile" it's like putting an addictive substance in front of an addict. Why chance it?
Is your dd the same age you were at the time he abused you? Why the sudden interest? I would be alarmed. It can take awhile for someone to sufficiently "Groom" a child and he may be in that process. If he hasn't gotten help it's unlikely he wouldn't repeat the behavior.
It's understandable though that you've had to make all sorts of concessions to grow up in that family. To whatever extent you can limit his access to you and your dd, I urge you to Get help doing that. You will heal and you don't want to risk your precious child being harmed by this man.