What HE detergent are you using? You may be getting build up from that.
We use Charlies soap powder or Country Save powder, about 1TBSP per load.
We dry pail with no baking soda, then a cold wash/rinse, then a hot wash, cold rinse. We don't use vinegar as we have soft water. About once every 3 months I soak everything in bac-out and wash them without detergent. I've been using pocket diapers for 2.5 years and this seems to work for us!
If you do have build...