Are you, possibly, overloading the machine? If "yes", then it's likely that all the detergent is not being removed in the rinse cycle. All your clothes should be able to tumble and churn easily in the machine.
You don't want to use fabric softener on your towels because they will be less absorbent. Fabric softener leaves residue that repels water.
This is what I would suggest trying:
1) Fewer items in the load
2) Warm water, if you aren't too concerned about the energy consumption; and,
3) A good detergent, one that doesn't leave residue (no optical brighteners, no softener added claims, etc. - these things are left on your clothes after the rinse).
4) Keep with a 1/4 cup of vinegar in the rinse.
Personally, I wash my sheets and towels with soap, which leaves them softer than washing with detergent. Soap doesn't clean as well as detergent on dirty clothes, but its more than sufficient for towels.
A lot of people on MDC make there laundry soap by mixing Fels Naptha soap gratings with washing soda and borax. I don't like Fels Naptha because I don't want to use a product with naptha, a petroleum derivative, and I detest the fragrance of this soap. But, mine, is a minority opinion on MDC.
I use Ivory soap when I've run out of homemade soap. Others on MDC have posted about using Dr. Bronner's soap, which I'm sure works fine, but it's fairly pricey. While you can do it by hand, I grate the bar of soap in my food processor. Then, I attach the steel chopping bade, add a cup of washing soda and 1/2 cup of borax and process until its all a find powder. About 1/4 cup in a large load of wash works very well. Depending on your washing machine capacity, you may need to add more or less.
Others here have praised Charlies Soap, which is available direct from the manufacturer online. I've also seen a few claims that it has been irritating for some. I've not tried it. Just know that it's not really a soap, it's a detergent - which is not a bad thing. The manufacturer claims that it doesn't leave any residue. If true, then it should leave laundry soft.
Hope some of the above helps.
Good luck.