It's hard to say without knowing exactly what is being said, however usually responses like this are caused by some previous event or events that cause you to be wary of it. It may help to analyze why you feel the way you do about whatever is being said. If this person is someone you otherwise get along with and you know them well enough to know they mean you no harm, it may be something you can talk about with them and explain what you're feeling and why. That part is optional and situational. Sometimes just understanding why you feel a certain way can help.

Sometimes its not what people say, but the vibe that they give off. I like to believe that there's a reason my mind naturally wants to avoid certain people while I can put up with others. There's just something about them and gods know what, but do I really want to find out

Dunno if vibes from typing are a true feel, but maybe its just something like that (plus what they're saying is simply annoying). Its relateable!