The idea here is that veganism hurts people by restricting access to clothes or food
Any time you don't do something immoral for personal gain you're restricting your access to that gain. You can't kill that guy over there and take his money - that restricts your access to money. I mean, what does that even mean?
If you refuse to purchase products made from slavery then that restricts your access to those products.
Convincing someone that consuming animal products is immoral and them thereby choosing not to engage in the use of those products is not the same as "restricting their access to" it. That's an inaccurate use of language.