you can still grow even with smaller richer more productive population, actually you are encouraged to be more productive with smaller population than vice versa

population growth ≠ wealth growth