[0] mentions “a typical Sun-relative 20–200 kilometers per second”, which would translate to 1.2-12 years for a 50 AU displacement, but only for stars that move 90° perpendicular to the sun; and for stars that are further away than Proxima Centauri or Wolf 359 you have to multiply with a corresponding factor. So I think it’s typically more in the range of centuries.

[0] https://en.m.wikipedia.org/wiki/List_of_nearest_stars_and_br...