Item — HN Reader

These things are hard, and I don't want to discount any of it. But three observations.

First, browser engines do reimplement most of this stuff, though that's been an evolution, it used to be they relied a lot more on platform text layout, for example. And for the stuff that's not completely reinvented (accessibility), they have a put in a lot of work to make cross-platform abstractions. In many cases (interfacing with the compositor comes to mind), browsers are the only viable open source code bases you can read.

Second, to a large extent OS platforms are stagnating. UWP was going to be a big advance over the old WinAPI ways of doing things, but it ended up being a dud. SwiftUI has performance problems for a number of reasons, including the fact they're still using software rendering for a lot of stuff, and their compositor model requires allocating huge amounts of memory for intermediate textures for all their UI elements.

Third, the business structure around platforms requires them to each do things differently, which creates gratuitous incompatibility. You really only need one high quality library to do, say, text layout, rather than having slightly different and slightly incompatible versions on each platform. (And very few places are gratuitous incompatibilities more frustrating than GPU infrastructure)

So for these reasons, I do believe it's worth exploring a new cross-platform GUI toolkit. It is pretty speculative, though; there are lots of ways it could fail.

Game engines are decent at a lot of this stuff, of which of course GPU infrastructure (and graphics rendering in general) is quite good. Most of them could use work in the text department though :)

2 comments

I think there’s tremendous benefit to what you’re proposing. The future I’m the most worried about is where every application ships its own bespoke, slightly wrong text layout and rendering engine. We see this already in video games and it’s a mess. (Which games allow you to copy+paste in and out of their chat system? It’s a crapshoot!)

The amount of work to do this well is high. Probably (easily) measured in engineer-years. All to reimplement something that - janky as it may be - already exists in some form in web browsers. But as you say, worth it if we can pull it off. But it’s a common good. The hardest part in my mind would be figuring out how to fund a project like this.

My ideal application platform isn’t a video game engine. It’s probably something like electron, but with most of the browser stuff stripped out. (Dump javascript, the game pad api, midi, usb, Bluetooth, tab isolation, and probably the DOM. And use Houdini or something instead of CSS). And then I’d happily swap the rendering for pietgpu or whatever will perform the best on modern platforms.

Basic desirada: Small and light, fast, cross platform and platform-native look and feel everywhere. Accessibility, IME, localisation, rtl- and ltr- language support, and so on.

Mobile is still a big question. But solvable if sufficient engineer time was poured into it. It’s just a big project.

How would a new cross-platform UI toolkit be meaningfully better than e.g. Jetpack Compose or JavaFX?

To me, the single biggest opportunity is performance. There's a lot that can be done if you optimize for that: plumb incremental computation through the pipeline from the app logic all the way to the GPU, run your logic in multiple threads, and just generally minimize the work done. I gave a talk[1] which goes into more detail.

Now, I freely admit that many applications don't need that much performance. But I think having a toolkit that is optimized along those lines could be a good basis for doing work on the other things (developer experience) that make a toolkit great.

[1]: https://www.youtube.com/watch?v=zVUTZlNCb8U

Thanks Raph! I'll watch your talk.

I think the Jetpack Compose guys have plans to multi-thread composition, and they already do it incrementally.

JavaFX has a reactive/observable properties framework where you can build chains of lazy computations which update only the part of the UI that needs to be changed, this is then propagated to a parallel render thread that runs concurrently with the app logic. They originally wanted to further parallelize the render thread via tiling, but never did so.

Performance problems obviously still occur, usually when trying to do complex layouts with lots of measurement.