It's sad indeed that anime comes with built-in hate. There are millions of people out there that think it's cool to hate anime, but they couldn't give you a single reason why they feel that way. They just go with the crowd mentality, and unfortunately the mentality out there is that anime is garbage. How we got to this point is another topic altogether.
That can make being an anime fan quite tough sometimes. Hopefully you don't let others influence you when it comes to your own opinions. Of course, you build those opinions from talking with others and reading up on subjects, but I'm talking about disliking anime simply because someone tells you its stupid. That's not anywhere near reason enough to consider anime not worth your time.
I have a feeling you're above and beyond that kind of influence, not just with anime but life in general. Viewing anime can be a very eye-opening experience, and it's usually open-minded people that join in on the fun. Although it doesn't seem like it from time to time, there are indeed very like-minded people out there that appreciate your passion.
Who's the one person in your life that just gets your anime interest? They don't question it, they accept it. Perhaps they even love anime right alongside you. Having that kind of support as an anime fan isn't something that should be overlooked or ignored. Sharing your passion with someone, be it anime or anything else, is a very important part of life.