The only time I feel perturbed by improper grammar is when it's a main point or topic of discussion, i.e a thread complaining about grammar. I personally cannot read a book, long text, or piece of writing in English (if I need to analyze them) unless it has no mistakes, or the mistakes are unnoticeable. That's just me, however.
I don't like it when people move to America but don't bother learning English. It's rude to go to a country and expect us to conform to you for your entire life because you don't want to bother learning the language. The first couple years, sure; you're still learning. Twenty years later? No. The only case that's fine with me is if you don't have to directly work with people or talk often. If you do, you're making it hard on other people.