My kids, especially my son, had a different reaction to cheese vs. milk, but it was the milk that was worse. As far as I can tell (looking back, since we're completely dairy-free now for different reasons) cheese was just fine, and I never noticed reactions to trace amounts of dairy in things, but his with 3 or 4 bites of ice cream (we never drank liquid milk) he vomited. Thought it was a fluke, tried again a month later, and again he vomited. I learned my lesson.
Do you consume milk products in larger volumes? On cereal, as ice cream, anything like that?
For us, we're dealing with an intolerance, I think. I have hope of healing, someday, but we're a few years away (my best guess right now).
ETA--this was stuff I saw when he was eating it, not when I was. Is there anything else in the cheese, a coloring agent, anything like that?