Milk comes from a smiling cow on a hill with a giant bell around its neck; a girl with two braids and a blue apron comes up the hill with a bucket to milk the cow under a rainbow. Why wouldn't the cow on the carton be smiling? I mean, so how could it be unethical with drinking milk today from a grocery store?
Actual milk production starts with bestiality and ends with a slaughtered calf. The cow gives birth and the calf is separated from the mother to avoid cutting into the yield and is either sold off to be slaughtered within a few weeks of being alive or... if it's a female in good condition it can be raised to perpetuate the cycle. The cow will continue this cycle, and will probably be killed before its sixth birthday, less than a third of the way through its life.
Given that the production of milk is inherently tied to the slaughter of animals, vegans argue (correctly) that lacto-vegetarianism represents a 0% middle ground between slaughtering animals and not. By drinking milk you are still subsidizing the slaughter of animals, and it doesn't matter if it's a "local farm", either.
You still are paying for cows to take food from developing countries, you're still contributing to the unsustainable nature of the animal agriculture industry, and doing a lot of ecological damage. Taking the milk and not the meat doesn't actually solve any of those issues, either. Cows and calves are slaughtered and abused for dairy.