The United States in particular has had many groups that abstain from something in particular or tried to ban, amongst other things, alcohol, tobacco, marijuana, pornography, etc. Vegans are no different from those people, just another set of religious zealots rolling through town trying to ban their thing. This is further supported by the fact that vegans typically sneer at half-measures like vegetarianism and also are just judgmental as religious Puritans.
We (correctly) ban indecent images of children, but would be hard-pressed to make the argument that this is somehow "puritanical" and that this labeling invalidates arguments against the exploitation of any children anywhere. So just blanket labeling of something as "puritanical" isn't useful.
All other religiously imposed puritanical systems that we sneer at have no clear undeniable victim. Take the banning of alcohol, which was legally successful for some time in the US: Alcohol only hurts those around you as a consequence of you hurting yourself. But carnism has a clear victim; there can be no debate that the steak on your plate required the death of an animal, and is therefore wrong under any moral system, not just a religious puritanical one.
There are plenty of things that are wrong and that we abstain from on ethical grounds that are actually completely legal. These things typically fall in the category of "being a dick". Convincing someone that something isn't right isn't the same as trying to get it banned.
Suppose you aren't convinced of the above arguments, and you still want to call vegans just another religious zealot. Ok great, you want to slap a name on the adherents of a position. That doesn't actually invalidate any of the ethical arguments against veganism. Animal cruelty is still wrong, and you've managed to avoid the issue by name-calling.