I was guessing pregnancy as well. My allergy symptoms to some foods (peanuts) increased after my second daughter. While I can't say definitely which it is for you, I would guess lactose intolerance. At least that's where I would start. Some LI folks can't tolerate any dairy at all, so just because you can't tolerate yogurt doesn't mean it's a dairy allergy instead.
BTW, I am "mildly lactose intolerant" (by which I mean the symptoms are not so bad if I stick with yogurt and severely limit plain milk products) and raw milk of any kind gives me excruciating GI pain. Just so you know that it's not the "wonder milk" for everybody!
Let's see.... what can be helpful signs to point to one or the other? I just can't think of definitive symptoms besides swelling and hives that belong exclusively to an allergy. Allergies are more likely to cause redness on skin esp. around the anus and odd mouth sensations (burning, prickling, tingling) but every food reacts differently in every individual and an allergy might act exactly like LI. Have you tried swiping some whole milk on sensitive skin, such as your inner arm and wrist?
Regardless, you need to eliminate it entirely, including casein and other milk derivatives in odd places. (Whey is a sneaker.... it's in everything it seems!) Then add back fermented dairy at first. I don't feel that lactose free milk is useful for anyone but those with the most mild cases. Mine's pretty mild and I notice the effects of it, so I have no idea about this product. Other's probably have some experience to contradict mine, that's why advice on this subject gets so tricky!