You can still provide very honest and valuable feedback when you exit a company, without being a jerk. If the move is really about you, and nothing that the company can or should change, then that's fine.
But if there are things that the company did to cause you to leave, or if a once great company changed directions and started diverting from what you considered an awesome place to work, then you are only hurting those you leave behind when you "focus on yourself" and not "focus on others or the company".
If you're exit interview is with HR, then it may very well not do any good. But if you're tactful and constructive with your feedback -- even if it is about things the company could change -- then it shouldn't hurt you. And if the interview is with someone above you -- or if it falls in their hands -- then it could very well benefit the company and the people you leave behind.
If I were your ex-coworker, I would be disappointed that you didn't take the opportunity to help improve the culture I'm still a part of. If I were your manager, I would want you to be honest so that I can improve the culture to attract better talent going forward.
If the company/people you work with don't handle constructive feedback very well, then that's their problem and not yours.
My advice: Be honest, but professional and constructive about it.