There's a strong argument that the knowledge someone has from being disadvantaged contributes to their views on justice. The original position also incentivizes ignoring the rights of small groups of people since you likely aren't going to be a part of one.
One also gets into issues like reparations for past injustices. Those benefit a small group of people at the cost of the many (hard to justify with the original position), but can still be a form of justice since we want to balance past wrongs.