Yeah, this is actually a super-complicated tradeoff. Whether layers are a win or not is largely a function of how much of the screen can be cached in the layer and how much needs to be re-rendered, so in the scrolling use case a function of how fast you're scrolling, and of course how long it takes to re-render the content. Once your scroll speed exceeds the rate at which new content can be rendered, the experience degrades.

My conclusion is that if being able to re-render the content consistently in under one frame time is feasible, it's a win over layers. Fortunately, this is possible if you're willing to write fast code to render on the GPU.

There's a lot more complexity to this based on whether compositor latency is optional. Generally if you go fullscreen you can bypass the compositor. This is feasible for games but much less so for terminals and editors. I think that in some cases on Windows a swapchain can be promoted to a hardware overlay, and there you also get the opportunity to save one frame of latency. I think we'll see more of this in the future.

I'll write about this in more detail (with measurements) soon.