If it even remotely has a hint of "goaty" flavor, then it's not being handled in a way to prevent it. It is very rich, think richer than whole cow milk from the store, yummy and easier to digest than cow.
Talk to and observe the farms milk handling procedures as well as knowing the animals are respectfully cared for and kept healthy. Make sure they wash udders, test(check) prior to milking in clean container for mastities or any other milk issues, by doing a milk squirt into a clean clear container and looking at it, (this is not always necessary, but ask how they watch for mastitis and other mamery gland health and decide for yourself if you are content with their answers), clean milking and immediatly after, filter and cool down.
It's the quick cool down along with clean milking/handling procedures that prevents the "goaty" taste in the milk, not necessarily having a buck in the same pen(although that can help in some cases to have the seperate). Personal observations of mine have shown this.
Ask for a taste of the milk, if they really wish for you to be a happy customer, they'll happily comply (or even offer before you ask).