Sounds to me, actually, like an intolerance to casein, the protein in cow's milk. My son was given formula as a young infant, and his gut bled. We were told it was casein intolerance, and that we should avoid all dairy for a few years and then try again, to see if he would outgrew the intolerance. It's not an allergy, not precisely, but it is serious because it can damage the gut. I would probably re-contact the doctor who advised starting dairy, and talk about this possibility.
me , he , my three , , and -- and the one we lost