I definitely react better to goat's milk than to cow's milk. I used it for a number of years without any obvious problems.
However, a few months ago I went on a very restricted diet (no grains, legumes, or dairy) for a few months, and felt better. When I had a little goat milk, I noticed that I didn't feel so great after drinking it- it was much more subtle than my reaction to cow's milk, but still it lessened my health. So now I don't use it at all.