When people make such comments, they don't seem to realize they're suggesting you don't like friendly animals.
As a cat owner, you may indeed have a cat hair on your sweater, just as you sometimes have toys in the living room when there are kids in the house.
People may think they're being funny, but it does seem to be a veiled insult.
If you ever hear this, it's tantamount to telling someone how to raise their kids or keep their house clean.
Perhaps the best thing to do is express sympathy; pet allergies can be a pain, but what else can you say?
Every cat owner knows domestic felines have a secret life that outsiders know nothing about.
It's a little-known fact that in households that include pets, babies have fewer allergies.