"Cartoons" is, in the west, used to refer to cartoons for little kids (ages up to 11), they usually have no meaning, it's not a media used to tell a story, just something to stream on TV so parents don't have to do their work raising their children.
In Japan anime is a substitute for TV series, covering all the generes covered by everything produced in hollywood.
So, Japanese don't need to distinguish "Cartoons" from "Anime" because they know animated shows aren't just for little kids. In the rest of the world "i watch cartoons, lol" is equal to "i watch looney toons and disney and i'm sooo gay".
Saying "You're too old to watch anime, faggot", is equal to "You're too old to watch 24, Lost, Heroes, (Insert your favorite tv series here), you faggot".
I consider much worse those people who get all excited at the next DC "omg superhero movie".