I see this as a huge red flag. It is clearly evident that you may not be marked above average for the next appraisal. Obviously being marked as irresponsible shows that the upper level is expecting you to work like a donkey!!!
I would just start preparing for my next move.
>> Changing job would be resetting all the good will and trust that I have built here.
To this, I can say from experience, that our job is to look for the next better job. While this is a bit sarcastic maybe, but really, as we know, that companies won't think of that goodwill when the next round of letting go people comes along. And going by your description, your goodwill is because of your quality of work, so if you plan/prepare/put the efforts for the next move while you are in this job, you will really be better off.
>> I don't really want to portray my peer as bad(He is mostly nice on the face),
Most likely and I think you imply politely that he is not nice really and he understands the game being played and is playing it. You don't want to play that game, which is fair enough, but you will lose as a consequence.
So overall, just my opinion, is that now this job is no longer the right fit and moving on seems the most sensible thing to do.