I make my swim dipes out of PUL and FOE. You want it snug enough at the waist and thighs to catch poo, there shouldn't be any gaps.
If you want to line the PUL, I would use suedecloth, just b/c it's so lightweight and would let water move through easier than fleece. I don't know, saturated fleece sounds heavy and clammy and icky to me. Gives me the willies.