Presenter here, feel free to ask questions.
Also thanks to the awesome Recurse Center for inviting me to speak and making the recording, and the audience for their great questions.
back
21 comments
I'm surprised you found creating attributed strings and CTLineRefs was acceptable. These are the main bottleneck in iTerm2's renderer, and I ditched them when rewriting the renderer in Metal. Since ASCII is an important fast-path for us, and other than ligatures in fonts for programmers there's no need to go through all that machinery. Did you do anything clever to get around the slowness of creating NSAttributedString's and CTLineRef's?
There's caching all over the place so that you don't end up redoing the expensive stuff (ie creating attributed strings & CTLineRefs). It seems to work pretty well in terms of maintaining 60fps.
Caching is great but it doesn’t help you when paging through a big file where the entire screen contents get replaced on each update. Is that not what you benchmark?
Latency and multi-process editing environment has been on my radar recently. Programming languages providing analysis servers, external syntax highlighters and of course REPLs all benefit from editor that’s fast and async. Since Jetbrains editors brought zero latency writing to my attention I’ve begun to notice which apps seem to hinder my writing. It’s getting easier to build your own editing system instead of having to accept full product from one vendor (Jetbrains tools are great, but very hard on my poor machine…). For this I’ll be watching Xi closely.
I’m in the process of learning Kakoune (1) which happens to have JSON-RPC API as well (2). Kak is fully terminal app which doesn’t give it any edge on latency (doing low latency terminals and shells seems to be a hard task). Hopefully Kak’s take on Vi’s commands as selection oriented language can be leveraged from within Xi editing framework in some capacity.
(1)[http://kakoune.org] (2)[https://github.com/mawww/kakoune/wiki/JSON-RPC]
Xi's medium-range plans for modal editing support[1] are intended to be modular/general enough that you could plug in whatever particular modal-editing implementation you'd like.
Thank you, that's a good thread. I see Kakoune is mentioned. Will have to check back later on.
I remember listening to your talk about Xi at RustConf '16, and being amazed at how deeply you've been thinking about text editors. This talk is a nice update on your thoughts.
How soon do you think it will be before more product-level thinking can be brought into the mix?
(Edit: I see I'm not the only one to ask a question along these lines.)
What kind of UX model do you have in mind? Something like the vi modes (command, visual, input, etc.), something more GUI like with input mode by default and hotkeys or something completely different?
Definitely the standard modeless text editing model (with multiple cursors though) as the default, and vi-compatible modes as an option. The issue for the latter is linked elsethread.
How powerful are you planning your extensions to be? On a scale from Textmate (shortcuts that run shell scripts) to Emacs (you can implement Vim in Emacs), where would Xi's extensions stand?
Considering plugins can be written in any language and communicate via JSON, I'd posit they can be whatever you prefer. Turing-complete, even.
Consider the following quedtions: Can they overload arbitrary key presses? Can they modify what is on the screen? Can they change the screen's content without changing the file's content? Can they call the editor's own editing functions? Can they be called by the editor's own editing functions?
Textmate can bind shell scripts to shortcuts, but letter keys always insert letters, and the editor always shows the file's content.
Emacs can execute arbitrary code on arbitrary keys, which enables you to implement vim-style key bindings. Also, Emacs can display stuff that is not a file, which enables mail clients, git clients, and terminal emulators.
Coming from the Emacs side of things, I will have to call that an incomplete understanding of the concept of extensibility.
Hi Raph. Do you ever get the feeling that `xi` will inevitably become isomorphic to the "general programming problem" that's been solved many times, at several levels of zoom, distinguished by trade-offs and implementation quality? I mean, what do you do when your general editor implementation inevitably starts feeling like an OS? Or do you think you'll be able to successfully avoid implementing processes, access control, roles, db and/or fs functions etc in your editor, somehow?
Can you please comment on Xi as a general editor and specifically in relation to vim and Emacs, arguably the two best general editors from the last few decades.
Thanks!
I don't want to knock either, they're both obviously great editors. My main goal is to build something that feels entirely like a native app. Also, I'm hoping that the async model for plug-ins will preserve responsiveness even when there are a lot of plug-ins and they do nontrivial work.
As an Emacs diehard, I'll be quite tempted to give this a spin once the plug-in model integrates nicely with some Lisp implementation (e.g. SBCL), and someone sets up a repository along the lines of Emacs Lisp Package Archive (ELPA). Cool project!
> My main goal is to build something that feels entirely like a native app.
It's not clear to me how this relates to the question. Emacs and vim are native apps.
For the protocol between core and front-end, did you take any inspiration from mosh's state sync protocol? https://mosh.org/mosh-paper-draft.pdf
Not specifically, but I did dig deep into the state syncing literature in general, both Operational Transforms and CRDT's. The problem we're trying to solve is a bit different than mosh, because we don't need to compensate for network latency in particular; RPC should be very fast.
Text editors shouldn't have a "core" and "front end" and "protocol".
Monolithic machine executable that fits into under a meg of RAM and comes up in a fraction of a second from a cold invocation.
Yes, they should. And here is why:
1. Stability. I'm writing my own editor at the moment, and the text buffers are kept in a server process. Last time I updated the server process it'd been running continuously for a month despite extensive reworking of the frontend. The backed is trivially simple (~300 lines), and easy to keep stable. If the frontend crashes it doesn't take my open buffers with it.
2. Coming up in a fraction of a second with state most of the time. Most of the time I have a ton of buffers in RAM; rather than reloading a ton of files, most of the time when I bring up the frontend, the files I'm working on are already there. Re-establishing an IPC connection to the backend is not noticeable.
3. Faster starts over slow network connections - I can run the backend remotely and not have to transfer a big file other than what is needed to view the bits I care about.
4. Simplicity: I don't need to implement tabs or multiple windows - I got that for free from my window manager. Instead if I want to split the current buffer, I just spawn a new frontend that re-attaches to the same buffer from another process. There should be no need for an editor to re-implement its own window management.
A client-server design makes very little difference to memory usage and doesn't require a separate executable.
You made a decision to focus on GPU rendering for now. Do you think that architectural decision will impede high performance software rendering as a fallback?
Hardware rendering routinely breaks in Ubuntu (and probably other distros) because NVIDIA's driver still isn't packaged well enough to survive a kernel update, and doesn't work over ssh or on an many embedded systems anyway.
Hi, good talk. I was wondering if you could elaborate more on your JSON RPC mechanism and the use of threads vs. processes. Coming from the Python world I could see it very hard or impossible to use one plugin written in Python2 and another written in Python3 within the same process.
I have successfully integrated several interpreters/event loops into the same process (C++/Boost, Python, Tcl/Tk, Qt). It was a real pain to implement and felt very hacky but worked.
Using processes you can achieve something like a micro-service architecture. An example of where I've seen this kind of coupling is the custom transfer agents in Git LFS.
https://github.com/git-lfs/git-lfs/blob/master/docs/custom-t...
They use processes and line delimited JSON for communication.
Nice talk!
I wonder, in the talk you say that you took some ideas from the design of Chrome; do you think that --in turn-- browsers could learn from the concepts in Xi?
And another question: you are using multiple languages. Doesn't that make it needlessly difficult to express the same operations in different (concurrent) parts of the system? E.g. a character update is rendered on the screen (in Swift), and simultaneously the operation is sent to the core (in Rust) to reconcile the change in the core data structures; both operations are essentially the same, but now they have to be written in a different language, which means more development work and increased likelihood of mistakes/inconsistencies (?)
PS: I'm not sure if this makes sense; I couldn't finish watching the talk but will watch the rest later!
I'm still learning the architecture but my understanding is that the way it works is that the frontend sends the character to the backend which incorporates it into the model & then generates a notification back to the front-end via the regular delta diff algorithm. The front-end then applies this delta to its data structures that represent the view & updates the rendering. All of this turns out to be super quick because the rope data structure applies changes quickly & the frontend is modifying very little data to update its rendering. My hunch is that it's probably less efficient if you're trying to do a lot of little modifications very quickly vs other editors but that's not a realistic use-case; you're usually inputting individual characters at human speed or doing large modifications via bulk operations that can occur in 1 delta.
Is there any particular reason json-rpc was used compared to more performant gRPC?
Universal out-of-the-box support in almost every language. The actual performance impact is subtle - for most core/front-end interactions, the messages are very small (because we put so much effort into minimizing the deltas). It's also the case that there are ridiculously fast JSON implementations out there. We did discover that Swift's is not one of them though. I have a prototype of a faster one using Codable; I'd consider upstreaming it to Swift but just need to find the time.
I'd consider a different serialization format, but it doesn't seem to be on the critical path for either performance or functionality, so I feel there are a lot of other things ahead of it.
Off topic, but wonderful job on Inconsolata!
Hey, stay on thread! Thanks for the kind words though.
Sorry to prolong the tangent, but that font is truly legendary. A milestone in the human quest for clarity and form, especially if that human is me.
Where is the Xi plugin documentation (as in, how to write and install a plugin, what are the APIs).
I found https://github.com/google/xi-editor/blob/master/doc/plugin.m... , but it doesn't have anything technical (or a tutorial).
It's still a work in progress, we're revving the plug-in protocol. I'm looking forward to the time we can point people to a stable interface, but sadly it's not there yet.
This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google
Would you mind elaborating on this?
not the op. just my understanding is that it is raphlinus's code. It means Google has literally nothing to do with it, other than happening to own the code.
(IE it's not an experimental product, it's not a product at all. It's just raphlinus releasing some code)
Would you say Xi is usable for "real-life end users"? If yes, what are the best-supported usecases? If not yet - do you have a plan for getting there, or is it still mostly a research project?
(FWIW, I tried compiling Xi on OSx & opening a large file in it. To its credit, it worked, but there are issues; e.g. horizontal scrollbar is wrong, word wrapping is not working etc)
Great work, and I'm looking forward to getting involved if I can. I'm just learning about this project, and am very interested as I'm doing a project with similar performance goals for filesystem navigation. It even has a CRDT based concurrency model, and a multi process architecture with back and front-end applications communicating across a JSON api.
I'm curious about your reasoning for JSON, as compared to protocol buffers, flat-buffers, etc. I would imagine that accessibility and ease of use would be the main reason for choosing JSON, is that in fact the case?
Are there any other reasons you chose JSON over other wire formats/protocols?
I agree with your assertion in the video that this is not much overhead, but I struggle with this debate as I work to scrape every ounce of performance out of the architecture.
Edit: Asked and answered elsewhere https://news.ycombinator.com/item?id=16268332
Most of the messages are tiny & the decoding occurs off the main thread (encoding varies I think). JSON is definitely fine as a first choice to get the system running but we are considering at some point upgrading to something more modern like Cap'n'proto or flatbuffers that offers free encode/decode.
Great talk, I've been eyeing Xi for a bit now and am really looking forward to giving it a shot. Is there any window for when this can actually be put to use on Linux/Mac? I've had a look at the frontends, but they're not very active and they're pretty bare so far. Any idea when these will be more substantial?
The xi-gtk frontend is in pretty good shape, but overall the editor is not _quite_ at the point where I'd recommend it for daily use. I don't want to make a promise of a particular date, but I'm hoping fairly soon.
Do you think the language used will last for 20 years in terms of developer interest?
You mean Rust? Yes. However (and I didn't get into this in the talk), I actually feel that if I figure out proper async _protocols_ that express rich functionality in neatly composable modules, then that protocol is likely to outlast any given implementation. So if in 10 years somebody figures out a way to make ATS-style dependent types easy to use, I wouldn't be surprised if there's a core written in that.
Why the name?
I wanted something short, and it felt kinda futuristic. The similarity to "vi" was also intentional.
(Not the author) I presume a reference to one the many mathematical/scientific usages? https://en.wikipedia.org/wiki/Xi_(letter)#Mathematics_and_sc...
I particularly like the "No change of state" meaning in "Z notation", apparently a formal language for specifying and modeling computer systems.
Edit: never mind, author beat me and debunked my overly-complicated analysis
How far away are you from a cross platform 'product'?
At the rate we're going, it will be a while. It takes time, and I'm trying to get the right answers rather than racing toward shipping. It's a very different environment than a product-focused startup, and I'm happy about that.