If you’re of a certain background, it’s really your mom or dad’s alma mater, and then one of a few “perfectly acceptable, fine schools.”
I think the difference between say #4 and #12 in a given year exists in the minds of middle class strivers. I went to a public university and a private one for undergraduate and graduate school, and at the top levels it really comes down to the professors on an individual level and perhaps the department, more than the institution itself.
And in any case, if you pay attention to the finer movements in the rankings, schools rarely keep the same position between publications. It’s just the same ones playing musical chairs with each other year after year.
If you got into #4 and #40, then yes, you were going with #4.