I consider myself a "normal, socially apt, law abiding person", and I browse /b/, watch animu, etc.
The reason a vast majority of like-minded people are fucking retards is because they have no perspective, or sense of scale.
They'll allow themselves to get involved in something like being furry, and rather than just leave it at "I have a very strange sexual fetish", they feel like they have to make it a life-style. The same goes for most of these groups we're talking about. Anime fans can't watch a show and say to themselves, "Wow, I really enjoyed the story I was told, and I think the characters were really neat.". Instead for some reason they feel compelled to dress up like them, start talking like them, and make stupid e-mail addresses.
I look at loli on /b/, or watch some cheesy moe shit, I enjoy it, RAAAGE, do whatever, and then I get off my computer and go back to normal life. There are alot of people that do the same thing, but you never hear about them; you only hear about the crazies.