I'm the same way w/cow's milk. Although I also feel very lethargic and heavy, get headaches, and my nose does not stop running. I'd have these lil' old tissue colonies on the side of my bed when I woke in the morning.
I can tolerate goat's milk no problem. I get raw goat milk from a local farm and make lots of yogurt and eat cheese and goat butter and have no problem. I hope this will be your experience as well