My son and I had markedly different reactions to straight milk (a glass, or ice cream) than to hard cheese (and we never ate soft cheeses like cottage cheese, so I don't know about that). We've ended up dairy-free for different reasons, but the difference was marked enough that I think, for some people, there is a difference (and eventually for us, we may be able to consume some dairy products but maybe not others--we've got a while before I test that out).
To really know whether any or all dairy is problematic, I'd say it _is_ best to eliminate all dairy, every last bit, and then be very deliberate in what you introduce. Some people can tolerate goat's but not cow's milk (these would be intolerances, not allergies, most likely--but all of our stuff is intolerances, which are quite real, just different), or maybe fermented dairy first (things like goat's milk kefir, which seem incredibly helpful on the gut), and then move on to other products (be careful with storebought yogurt, almost all the brands have more dairy added _after_ the original fermentation, how weird is that?). By taking it all out for a while (I'd say at least a couple weeks, a month would be better--my kids and I are fairly quick in getting this stuff out of us, so I'm not good on timelines), you won't be as likely to miss a low-level problem.
