I think the idea comes from many other mainstream languages being relatively much faster. The classic rejoinder was about expressiveness and developer time, but with newer languages that have learnt from Ruby and others (most obviously Elixir and Crystal, but any of the newer generation could probably be chosen) that's less of an argument. Now it relies on the legacy of Rails more than anything, which isn't so bad, is it?

I wouldn't pick it for a new project though if I could write Rust or Elixir or Crystal etc unless they lack something in their ecosystem that Rails has, but over time this will become less of a difference.

No, it is actually from even before that, from Ruby 1.8 which did not have a proper VM.