You can't control other people and what they think and how they act.

You can control yourself, however.

So, you don't get them off your back...

You just ignore them and put zero value in their opinions.

Lol. "ur gay", if I had a nickle for every moron on YouTube that's called me that I'd be a wealthy man.

Inform them that there was a study that showed that people who call others 'homosexuals' in a negative context are more likely to be gay themselves. :3

Sing 'Picture to Burn' verbatim when others could hear you. Really, show them (1) You don't care what they think about your taste in music and (2) You don't find being accused of being gay as an insult.

BTW, I'm a 40 year old dude who also likes Taylor Swift. Am married, have a kid, and will sing 'Picture to Burn' verbatim whenever I like.

first, don't say "a gay" as that sounds offensive.
second, just ignore it. so you like taylor swift. you sexuality is your business & they're just trying to rile you up.
the best way to make them stop is to show them it doesn't bother you, even if it secretly does.

Beat the shit out of them!