I'm working on cuttting all dairy out as well. I don't have anything to back this up, but I believe that dairy(cow) is the #1 sensitivity in babies--like if you're nursing and need to start cutting things out of your diet you try dairy first.
I'm not sure how it plays out later in life. I didn't have this problem at all with my first, but with my second he would get horrible diaper rashes if I had milk/ice cream but not cheese or yogurt. This one (#3) just seems to be much more sensitive to my diet.
Honestly, we're just not meant to eat dairy. Personally I know it's just not a tolerance that my father's line has. I've seen a chart breaking down traditional foods by ethnicity, and I'm just not from a dairy tolerant group.