On the other side of the argument, yeah many things are imaginary but that doesn't invalidate them. A football team is an imaginary group but serves a purpose, to play a game. Country borders as well separate cultures, food, music and people with different languages, that's fine, these make sense, but why isn't free movement allowed? Polish didn't move to Germany en masse nor Germans to Poland while there is no border control and there certainly is economic inequality between these countries.
The part that it will be messy, was actually what yourself said (so I didn't think you need evidence for that).
Well, one can also look at other models historically, if you think of a border as demarcation of the influence of a state. Historically China didn't have well defined border, the further away civilisations were vassal states which had to pay tribute (but received gift of higher value than the tribute, so the real tribute was acknowledgement of power).
Another example would be afghanistan/Pakistan. The border is very transparent and at least in Afghanistan there is not a functioning government, but war lord tribal areas.
But this is about border of state demarcation. The idea of people moving to other places is generally a new one (apart from earliest pre history or when masses do this, like in the migration period) new in the sense starting with colonization of the US.