It's going to take time. Dryer sheets cause a build-up inside the dryer. I'm allergic to the stuff, so we have never used them, but my mom uses them in her dryer, which we use at their house (once or twice a year). We use vinegar in the rinse cycle - maybe that's one reason why I don't notice it there?
If it were me, I'd get on my knees and scrub the inside with a good vinegar cleaner: 4 thieves comes to mind, but just vinegar/water would work - or vinegar with hydrogen peroxide (this is an awesome cleaning solution - get 2 spray bottles, one normal white vinegar, one normal hydrogen peroxide - and you spray each of them on your surface and wipe off - you can't premix because they will lose their potency) ... take out the lint filter and clean it, too.
Then, I'd probably wash/soak some towels in diluted but strong vinegar and dry them on high heat. I don't know if that would work, but it seems logical to me!!