I have never been so happy to find something wrong with DS. Turns out he has a posterior tongue tie! It totally explains why things started to get better after I went off dairy then started to get worse and worse. I never noticed the tongue tie because I had such a healthy supply to start with he didn't have to work. I also never really had anything beyond minor discomfort when I started breastfeeding - turns out I have incredibly stretchy nipples and a really high pain tolerance!
I have a consult tomorrow to get it clipped and hopefully can have the procedure done soon so I can start working on building back up my supply. The lactation consultant made me feel really good about the situation - I've been putting in 150% doing breast compressions, pumping constantly, taking all kinds of herbs but he has only been able to do 50%.