I'm a veterinarian in the US, and love my job, but like every career it has its difficult moments. You most certainly have to be good with people, not just animals.
First off, I'd say its a difficult path to begin with. You have to be very good at math and science. It is harder to get into vet school than med school. The people who come in as clients and say "oh, I was going to be a vet, but I would get too sad" just make me wonder if they have any idea how rigorous the requirements and education is. My husband is a human MD and PhD, so I was able to compare his experience to mine... and they are vastly different. In order to be a vet you have to know every system of many species. Human side medicine is far more specialized. Human med school is also more expensive, but vet school isn't cheap by any means.
The economic reality of working as a vet is comparable to working as a dentist. Since the client isn't insulated from the cost of health care, the whole experience and choices made are different and require both flexibility and tact.
I would also caution you that people assume that "If you really love animals, you'll save this one for free". They don't always understand the economic reality of running a business, and the demands can be challenging when dealing with people who want something for nothing. If the general public didn't have the idea that a DVM should be altruistic or that they should dictate when and where I do my charity work it would make life as a practicing vet far more enjoyable.
All that said... its the best job I could imagine. Yeah, everyone loves the well puppy visits, but the feeling that you get from fixing an animal so that they can live a better life is huge.