The idea here is that the only individuals that deserve rights are humans. The other way this is stated is to simply deny that it is an ethical issue altogether, or that animals have no moral value intrinsically.
We grant plenty of animals rights. Dogs have a right to not be forced into dog fighting. Unless a carnist is willing to step forward and say that they are okay with dog fighting, bestiality, poaching, and any form of animal torture, then they don't believe this. If they do, then would they rather hire a babysitter for their children that is known to torture animals in their basement? Or would you really say that if a baby dolphin approached you on a beach, it really be no big deal if you cut its fins off and threw it back in the water? I'm sure that if you did that most of the beach-goers would be talking about that incident in therapy two decades later.
The thing is, most carnists reveal this isn't their actual position when they say things like "of course factory farming is bad but..." or "humane slaughter" or "it's not like I'm into horse racing where the animals truly get abused". Any of that talk means you don't actually agree with this premise because you believe that at least some animals have some moral value. Therefore, your argument isn't that humans are the only things with rights. Again, you would have to then with a straight face agree that if you saw someone holding a dog between their knees with a belt around its mouth digging its eyes out with a rusty spoon as it struggled in pain you would essentially respond with a "meh" and a shrug and would be unable to identify that as problematic.
Therefore, if you (like most normal people) do indeed believe that some animals have rights, then you would need to justify why it is that you assert specifically some animals that specifically are allowed to have their well-being harmed in specific ways, or why you assign morality on a species-by-species basis.
Let's press on assuming that this position is not merely being adopted because it's convenient (e.g.), but actually is a position that genuinely reflects someone's real views. You would still need to justify what it is about humans that makes them different from other animals, and in particular, whatever you use had better capture all the humans. So if intelligence is your criterion, you'd better be okay with eating the mentally handicapped (Since most people aren't that's a bad one). This is the first hurdle that you have to cross.
We can try to get to the second which is demonstrating that this isn't an assertion pulled from nowhere out of convenience, but most criteria never make it past the first step without some blatant special pleading, such as just asserting that the mentally handicapped are, for example, magically "grandfathered in" or included by some other arbitrary reason.
It's worth noting that animal agriculture destroys the ecology of areas surrounding factory farms, leads to global warming, and starves poorer countries of their food, while forcing non-participants to pay for it with animal agriculture subsidies (read: theft), amongst other issues. So yes, it hurts people. You should be concerned with such human rights as well.