An X server is a fantastic example of how building drivers for deep hardware components should never have been written in userspace.
(If anyone wants: There's a fantastic talk at LCA a few years ago from Keith Packard about X's history: https://youtu.be/cj02_UeUnGQ )
It also has lots of flaws. "Mechanism, not policy" has to go, because you are trying to make a desktop environment, not draw arbitrary rectangles to the screen. It is already a given that there is one special client called the window manager; it would serve the architecture well to know that's the window manager instead of treating every client as if it could potentially be a window manager. But Wayland goes even further in this direction, eschewing things like icons, title bars and resize borders, which are basic expectations of a desktop environment. While X11's way of doing these things is esoteric, at least it has one. (If it were up to me, standardized window properties would be promoted to requests, so you'd just call xcb_set_window_title instead of fussing with atoms and window properties).
Another anecdote is HDR: Wayland seems to still have not standardized how it should work; meanwhile on X11, there is already protocol support for multiple pixel formats (1-bit, 4-bit, 8-bit, 16-bit, 24-bit displays) and you could just add 48-bit color to that list. That would be one of the things Wayland stripped out while striving for simplicity (since everyone had 24-bit colour at the time) that turned out to actually be important.
---
At some time before the heat death of the universe, I will get around to trying to fork Xorg and remove all the cruft and see where that goes. (Server-side rendering is not cruft. User-mode SVGA drivers are.)
I did minimal work on this already - as in, I downloaded Xorg's source code and had a look through. The whole configuration system has to go. The XFree86 driver code and driver abstraction has to go, and the DRM driver hard-wired in place. Using dlsym to locate compiled-in modules makes tracking dependencies impossible.
I suspect this has something to do with the state of Xorg: Nobody knows what it should and shouldn't do, and the module API for Xorg is the entirety of Xorg, so changing anything is liable to break someone's external module that wasn't on the "it should do this" list.
It wouldn't be the first time the X server was forked with the intention of cleaning it up: Xorg is a fork of XFree86, which was a fork of X386, which was based on something I don't remember.
However, there's just stuff that X assumes that isn't true anymore. A good example is how it handles input. How many cursors does a screen have? One machine? okay, One input device?
One of the folks behind both Wayland and X gave a fantastic talk on the issues and the hell. https://www.youtube.com/watch?v=HllUoT_WE7Y
This looks so obvious that it ought to have been tried.
If I am ever forced to use Wayland (yes forced), I could very well seem my self using Compositor.
Great work!
I seem to recall WindowMaker had special support for gnustep's NeXT style detachable menus.