If the performance hit is seen as the overhead of the actual object oriented pattern (assigning fields to objects, caching, etc), then yeah, not that much slower, after all, the allocations would have to happen at some point. Patterns like DataMapper pattern present an alternate path that gives ergonomic advantages while not hiding the hard parts of dealing with state AND being incredibly performant.

But the real performance cost of ORMs has always been what typically gets left on the table: well written queries and schema. Poorly written schemas inspire poorly written queries which in turn completes the ouroboros by inspiring poorly written indexes. And ORM's like ActiveRecord encourage this. One easy example of this is the Paranoia gem as an addon to ActiveRecord, which encourages the misguided pattern of keeping around old data in your RDBMS.. There are many more examples.