I get pilly-ness too, and it's frustrating. I use fleece liners, and they aren't soft on the skin when they're rough from the pillies! I just tossed a bunch and made more, but from microfleece, and I think it's more resistant that the polar fleece.
Maybe for the FBs, you could buy an electric fuz buster thing and buz off the fuz every once in a while. That's the only thing I've been able to come up with.
And is it really necessary to wash anything with fleece separately? That sounds like such a pain!