It is somewhat easy to forget for a casual player (such as myself) but I'm not sure that it's really unfair... it seems to occur just infrequently enough that I don't always take advantage of it. But when my opponent does it to me and I wasn't expecting it, I see it as my own fault. It's simply a rule of chess.
EDIT: I see now the title has been changed to "En passant — a lesser known chess move". I replied when the title said it was an "Unfairly unknown" move