My son's behavior is profoundly different on and off milk. Lory (wolfmeis) has seen this first hand, as this issue darn near spoiled a weekend get-together for us.
He was not sensitive to dairy, as far as I know, when he was smaller. It has crept up on him over the last year or so (since he was about 3? Maybe starting when he weaned but picking up speed later?) and now at 4, if he's had dairy you can tell by his behavior alone.
The basic difference is lack of inhibition, but I think that this stems from an overall grumpiness and general po'dness that comes with feeling like crud. The difference is that off dairy, he'll respond to me when I call his name once (sometimes a couple times if he's really involved) and ON dairy, I feel like I'm screaming his name all the time to get him to stop what he's doing and listen to me. Off dairy: cool, lets pretend sword-fight. On dairy: well dang, it is just so FUN and SATISFYING to hear you scream, big sis, when I whack you with this stick, that I'll do it over, and over, and over, while my mom screams my name from the patio.
Ugh.
Since cutting dairy from his diet, the hurting-people-on-purpose issue has dissappeared almost entirely (he'll still haul off and whack his sister one during an arguement every now and then, but it is in normal kid proportions). I don't have to worry at parks and playdates anymore (he's still a huge kid, and sheer size makes me nervous sometimes, but his fuse is way longer and I don't worry about him hurting other kids out of the blue anymore).
In short, YES, dairy may be causing your toddler's problems! My son's symptoms are PURELY behavioral (as far as we can measure - I SUSPECT he also gets headachey and PMS-style grumpy but he says he feels normal) and ENTIRELY related to the dairy.
Sorry, I forgot since it has ceased to be an issue - there IS a physical symptom for my son! His hearing was only so/so and he got frequent ear infections and blockages until we cut dairy from his diet. If he has a bunch of dairy (just a little won't do the trick) he also can't hear as well for a week or so. Cutting out dairy would have been worthwhile for this reason alone; we had to yell for him to hear us, say, from the back of the car. This symptom, though, is not visible or measurable the way that bloody stools or hives would be.
He has also stopped wetting the bed at night. This could be due solely to the fact that he is getting older, of course, but the rapidity with which this behavior disappeared, along with the waking up disoriented and wailing, makes me suspect it is allergy-related as well, especially after reading online that dairy allergic kids are more likely to wet the bed to later ages.
There are a host of things that removing dairy from his diet has solved for us. When Lory (wolfmeis) suggested that this might be his problem, and I read up online and saw that almost all of the behavioral and medical issues I'd been pulling my hair over were known to be related to dairy, I was thinking; DANG this is just TOO GOOD TO BE TRUE. But it really has solved so much for us, and has turned my son from one of those kids that raises the household stress level to the point where you ask on a daily basis: WHAT am I going to DO with this kid! to a 'normal' kid with normal issues.