There are plenty of counterexamples to this. Consider Finland, where most of the population is comfortable with English and admirably grammatically correct (though they never really lose that distinctive Finnish accent), but they have never lived in an English-speaking country, and in fact Finns travel in general less than their Nordic neighbors. Learning English does not seem to strike the Finns as “hard”, and if you ask them about the process you often get the usual claim that English is an “easy language”.
Looking around European nations, the key to good English seems to be 1) English in school taught from an early age and with a serious approach, and 2) subtitles on television and films, never dubbing (or, as in the case of Romania in the early millennium, a very active torrent community downloading American releases).