Committing to Rust for Kernel Code

I start taking LLVM serious when they give easy to follow documentation on how to compile LLVM completely GCC/binutils/gnu libc independent. I mean a single go to point with easy instructions.

You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point. Fucking make that straightforward already dear LLVM project.

I've been compiling my kernel on Gentoo with LLVM for around a year (since I moved to Linux Desktop) and I've had 0 issues. Granted, I run a very stripped down config, with almost only related to my hardware enabled, but I run the latest unstable (not Git) kernel, 6.6.2 as of today.

Edit: forgot to mention, ThinLTO from 6.6.2 :)

There's something incredible about rust in that once it clicks, it results in profound changes in how you are thinking about your code. I know it has certainly improved mine in numerous ways.

What have been the main lessons from your C to Rust migration? Was there anything that surprised you?

C# is among the languages that influenced Rust's design :)

1, of course.

Bugs can be fixed.

Bit of an aside, but these are not the only two options.

It's true that Rust is currently the only stable language in this space, so it has some advantage there, but "Rust or forever C, there are no other options" is of course a load of nonsense.

According to pcwalton appears avoiding monomorphization on its own may be unlikely to yield substantial gains [0]:

> I doubt that a hypothetical version of Rust that avoided monomorphization would compile any faster. I remember doing experiments to that effect in the early days and found that monomorphization wasn't really slower. That's because all the runtime bookkeeping necessary to operate on value types generically adds up to a ton of code that has to be optimized away, and it ends up a wash in the end. As a point of comparison, Swift does all this bookkeeping, and it's not appreciably faster to compile than Rust; Swift goes this route for ABI stability reasons, not for compiler performance.

> What you would need to go faster would be not only a non-monomorphizing compiler but also boxed types. That would be a very different language, one higher-level than even Go (which monomorphizes generics).

[0]: https://news.ycombinator.com/item?id=38224941

Compile times have greatly improved over the last few years. And if you aren't importing 100+ dependencies from Cargo.toml or using a lot of templates/traits (which I suspect kernel code won't), then it's even better.

Edit: Also kernel compile times are a problem for the elite few, while kernel bugs are a problem for everyone.

Compared to C, yes. However the tradeoff you get is tighter iteration, with more errors being found at compile-time rather than having to wait for runtime.

If you have to restart an embedded system or schedule a job on a cluster to properly test, that can work out to huge savings.

It would still be nice to see them substantially improve. I know there’s been effort to get debug mode to build substantially faster with cranelift, but I forget if that’s in stable now.

What are you compiling? I've "cargo install"-ed - which means deps had to be built too - two rust tools today on a 4c/8t laptop and they were done in maybe two minutes each. Building AUR packages of some C tools isn't much faster in comparison.

It's way faster than C++ - doubly so if you count "time spent grokking a compiler error" - and if that's the other option (it usually is), I'll take Rust

True, but I've been highly bullish on that: if that's a core complaint, it seems one of the easier ones to improve in the medium term.

A host of other issues, which Rust notably doesn't suffer from, are almost "unfixable", even in the long term: think build system, type system, memory safety. Rust is fine in these regards.

I haven’t done any Rust-on-Linux development, but it seems like this would be a non-issue there? A lot of the compile time issues for Rust come from cargo packages pulling in tons of monomorphized types, but I assume Rust-on-Linux has few-to-no dependencies (I’m pretty sure they don’t even use the standard library).

How much longer does `rustc` typically take to compile the same programming tasks compared to `gcc`? Is it just "a bit" e.g. 50% slower or by an order of magnitude?

Is there evidence that Rust compiles any slower than C++? Granted, it doesn't compile at the speed of C, but it does a lot more and you can write a lot denser code, so one line of Rust may easily supplant dozens of lines of C.

The only problem with Rust is that it has ALGOL syntax. If it had Lisp syntax it would be the perfect language.

I think you have misunderstood rust.

Rust is not fundamentally designed to work at a higher level of abstraction than C. Rust is designed to work at a range of abstraction levels, from as low as the lowest level C code to something fairly high (but still short of a language like python). Rust is designed to make it easy to quickly build abstractions on top of super low level (rust) code, because usually that's the less error prone way to do things - that doesn't prevent it from working at those levels though, and if anything it makes working on systems where you have to work on super low level code much easier to use.

When working with low level rust you don't "program everything in unsafe mode", you program a few really low level bits in unsafe rust, but quickly make abstractions that allow you to avoid making the vast majority of your code unsafe. This is opposed to languages like C/C++/zig/... where there isn't anything but "unsafe".

Rust doesn't really have a runtime by default (beyond libc for non-baremetal code).

There are a few tiny niches where rust isn't designed to compete with C, but they really are tiny niches. For example where chars aren't 8 bits.

I'm honestly just not following what you're saying about C++. Rust code tends not to depend on C++ code apart from maybe some system libraries that everyone depends on. The rust community tends to be overly-keen on rewriting those libraries in rust, not underly-keen. Obviously some people will make code that manages to depend on the language - but rust doesn't make it easy (like zig does...).

As for the relationship with llvm - rust doesn't actually have a hard dependency on it anymore (with support for a compiler backend called cranelift, written in rust). I also just don't see "the compiler depends on another language" as a huge impediment - as others have mentioned C compilers tend to as well...

> Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about?

Not likely. At the moment you need to do extra work to get Rust working well. It's not exactly beginner friendly and doing work in the kernel, you'll need to dig into C anyway.

> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?

Serious code already exists, so... Yes?

> Do you just program everything in unsafe mode? What about runtimes?

Why would you? You need that only when interfacing with something that can't hold the Rust compiler assumptions. See for example https://github.com/AsahiLinux/linux/blob/gpu/rebase-6.4/driv...

The few places that need direct access / unsafe are almost all single-line areas with an explanation.

> but it ain't the C culture if we are to accept that such a thing exists.

I don't get that whole part of what you wrote. What seems to be the culture / why does it matter in the technical sense? Some parts of the compiler are written in C++ - what's the specific issue here?

Having a memory safe language in the kernel is very serious. More and more C programmers I speak to say they realise with Rust they will likely never work with C again full time.

Yes Rust helps bring in the next generation of Linux devs. It needs to to survive. But it's not because young people like Rust, it's because C developers like rust too.

Yes, Rust has unsafe and unsafe rust is trickier to get right, but you use it opt in and it's only a small amount if the code. Zig is still having unsoundness bugs in relatively simple code

"Not having C++ anywhere in the stack" is not a goal of the C language or any major implementation I know of. The two major OSS C compilers, GCC and Clang, are both C++ codebases.

I tend to agree with your view.

I attempted to port a low latency trading system from C to Rust last year, but ended up abandoning after 6 month of development.

The overall feeling that I got is that Rust made my life a dream for 80% of the codebase, the _non critical, non high performance part_.

There are tons of well written libraries, cargo is awesome, the performance is on par with what C++ would do, etc.

But the last 20% of the code base, the high performance part, was a nightmare to implement. The Rust static safety basically get in your way constantly for any kind of non obvious memory layout (self reference, etc). Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path, and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.

I keep some resentment against the Rust community from this experience to be honest. I felt like instead of understanding the constraints & limitations of a fixed, microsecond time budget that I had, and trying to find solutions and be open about possible improvements, the overall trend was more in trying to defend Rust with whatever it takes.

> is still mostly dependent on C (even C++ really, due to LLVM/Clang)

I'm not following the assertion that compilation with LLVM/Clang introduces a C++ dependency on the language being compiled (Rust here).

To focus on the C/C++ comparison. Rust very much leans towards C++ side of things if you go the std library route. It does bake in some abstraction which would not be appropriate for low level systems development such as the Linux Kernel, which is also not what they're going for.

Rust gives you the option of opting out of that `#[no_std]`, which removes some basic tools, such as allocations, panics etc (I haven't used it much, so this may not be fully precise). As such with no_std you do move closer to what you'd consider C development.

That said rust will still have its restrictions, so some of the wild things you can do in C, isn't possible in rust, at least not without really violating the inner workings of it.

But with unsafe you can technically do most things on the level of C, without the overhead, that C++ would provide, via. its abstractions.

I'd expect the Linux Kernel to move in the direction of developing or using their own libraries and data structures, and that becoming its own bespoke part of their use of the rust language. Such that rust only really provides its safety guarantees and ergonomics. Compile times are also quite quick with no_std, as rust only really struggle with compile times, if you have lots of dependencies.

Circling back:

Rust is more like C++ with std library enabled (default) and C with `no_std`

It's basically a part of the larger project to "Rewrite It In Rust" -- to replace C code with Rust code wherever possible. C is dangerous to work with; and with safe Rust (but not with Zig), entire classes of bugs that have dogged the industry for decades simply go away.

> Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users.

Yes, the Rust language needs a complex optimizing compiler to really be viable. In this respect it is more limiting than ANSI/ISO C. For a lot of contexts, this isn't really a problem though? You aren't going to be able to build the Linux kernel without GCC or Clang anyway…

> The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C,

Yes

That is the point

Those abstractions are available to the programmer and compile down as tight and efficient as C

gcc is programmed in C++ too. The language of the compiler doesn't matter. What matters is C makes it too easy to make memory errors which Rust protects you from. This is why Linux should migrate to Rust and you can see this migration starting with adding support for Rust based drivers.

This was my impression as well. I haven’t spent enough time with Rust to hold a hard stance on it but it definitely seems like it was not designed for “bare-metal” applications. I hadn’t heard of Zig before this but I’m definitely going to look into it now. Got any recommendations for a good intro to zig?

> Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does.

I agree. In general I think Zig makes lot more sense as a C replacement for the kernel.

However, Rust is stable, and Zig isn't. And I don't expect it will be for some years. So at this point Zig is basically unusable for kernel work.

That doesn't mean it can't be added in the future though, and perhaps even overtake Rust. See: https://news.ycombinator.com/item?id=26815950

>It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023.

Like?

What other you guys do in C that isn't just integer oriented programming (using ints for everything)?

> And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case

Oxide Computer has written an embedded bare-metal OS in Rust, to serve their hardware product: https://hubris.oxide.computer/

I've been writing a little hobby operating system for the past 3 years in Rust and my impression is that the language is fine for low level code. The language improved a lot over the past decade and I think it's ready to be used more.

I've heard good things about Zig but I've never tried it so I can't compare unfortunately.

>The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang).

I think this might touch on part of the problem. Rust can work on higher abstractions but rust doesn't have to. It is a matter of the code base. I've worked on some that felt very similar in abstraction to C code bases. One in particular was a high performance nginx module. Most of the rust code interfaced with nginx directly through zero cost FFI interfaces to the nginx structs.

I've also worked on extremely meta, high level code that was written by a former Haskell dev.

Like many languages that can serve dual roles, the styles tend to adapt to it. You see very different styles of Java when looking at spring web apps or embedded android systems. Every language you have to pick and chose parts of it that, build a coding standard and need to enforce it. C is exactly the same and also has several roles people pick it for. The style of code of embedded systems, high performance networking systems and codebases that prioritize highly portability tend to be different and use different idioms. I think any of of us who have worked in a C code base that falls in one of those camps can name banned idioms that are normal in the others.

>> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.

My career before Rust was very C heavy. I've work from OS-less embedded systems, to device drivers, to Android porting. I've been doing Rust professionally since 2016. This just isn't my impression. A lot of folks I've known from college and internships who's careers went a similar direction ended up in rust independently. There is value in having a language that let's us get the job done that we used to do with C but with more high quality static code analysis. The business case and use case is very clear.

While I haven't done any bare metal development in Rust, in the areas I have worked I've found compiler, community and crate support to be great. I don't know where you are getting this impression. I think if I only read hackernews articles about Rust I would get the impressions that is Haskell that can work on bare metal. I've noticed articles that tend to get upvoted here tend to be either about 1) how quick adoption is 2) higher levels of abstraction like HKT or GATs 3) WASM 4) writing web services. This article is a great example of 1. 2 is catnip for this crowd. The majority of rust devs never need to know what a GAT is but if you used HN as a metric you'd think it was a daily problem. 4 is something that is easy to write, makes good content, approachable but isn't necessarily Rust's best use case. I'm not saying you've only been reading this site about Rust, just saying that it is easy to get a skewed impression of the language unless you really dive in.

> If there are any Rust experts around...what am I missing?

Probably more good examples and possibly more ecosystem.

A first big decision is if you're going no-std or not, and if you end up in the no-std world the ecosystem shrinks substantially. If you're building a kernel that's probably not that much of a problem - the same shrinkage occurs for C/C++ - many such projects bootstrap with nearly zero ecosystem anyway.

The examples side is a bigger problem - I've recently been able to watch some of my more curmudgeonly C friends give it a good dive, and after an initial hump they're fairly happy with the core language. They still have regular issues with the ecosystem when they run into, in their words "web dev crap", which comes up even in the stdlib sometimes - a bugbear a while ago coming up down some error handling code paths. They attempted to send patches and hit nebulous arguments against the correctness target, which were largely born of misunderstandings of posix. This kind of thing can come up anywhere, if you take a dependency on some fancy IO abstraction that happens to be written in C, and you take it somewhere "novel" like a BSD, you might well run into the same. The point here is that _examples_ and _exercise_ of these tasks are the things that are going to shake more of this kind of thing out. At the same time though, it's important to reiterate that if you're on the nostd path, then largely you're on your own, which is equivalent to just gcc bare, and this kind of thing generally doesn't come up.

> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?

This is a serious proposal. The outcome is really strong along key axes of correctness and safety. Those of us who've done it (e.g. Fuchsia, where I was) have been able to observe these benefits relative to history with the same teams using other languages (C, C++). We're professional engineers, these statements aren't coming from a place of craziness. The Android team have been writing about their journey: https://security.googleblog.com/search/label/rust

> Do you just program everything in unsafe mode?

Absolutely not. A good amount of bootstrapping effort has been going in across the ecosystem to make it ever easier to avoid unsafe. To take one slice of examples, there's crates that are designed to help you avoid copies while also avoiding dropping to unsafe - they provide tools for automatic structural analysis of the mapping boundary to make it easier to assert the relevant guarantees. Examples: https://docs.rs/zerocopy/latest/zerocopy/ (came out of Fuchsia), https://github.com/serde-rs/serde/releases/tag/v1.0.0 (serde is commonly used, but has more constraints here), https://rkyv.org/rkyv.html (not sure of prominence, but I hear people talk about it).

These kinds of tools get you a long way toward substantially safer code, without needing to think or audit nearly as much. We know that's important, we have plenty of data that demonstrates how important it is, and lately now, we have data that shows how effective it is too (see the aforementioned Android posts).

> What about runtimes?

They're out there, it depends what level of abstraction you're looking for, runtimes means different things to different people. For embedded there's typically a lot more focus on providing libraries rather than a whole runtime framework, so there are crates for a number of soc types out there which are well used, like https://docs.rs/cortex-m/latest/cortex_m/, or it's sibling minimal runtime crate https://docs.rs/cortex-m-rt/latest/cortex_m_rt/. As you get higher level, if you want to do more of the systems level interaction yourself there are a good number of options to choose from along the lines of reactor systems to get you to functional async executors that will build with nostd.

> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.

The challenge here is that this is an inverted view - it's a C programmers view of C++ being ported over to Rust, and it distorts how the world looks. That doesn't actually apply to the other sides _intent_. Yes, Rust provides a lot more abstraction capabilities than C does, and in that specific regard it has some coarse similarity to C++. It's definitely _possible_ for someone to way off the deep end and produce obscure abstractions around things that a more reductionist bias is going to hate - and you can totally ignore those things and have a great time with the language. There are some really nice things in there which anyone will enjoy if they come at it with an open mind, things like enums and pattern matching, the rich and efficient iterator library, the crates tooling, configuration macros, and so on. There's a lot to love, and they're not things that C++ did well, and comparing it to C++ discards those considerations spuriously.

> Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does

Zig is interesting in it's own right, and on a very surface level it is more similar to C, though this surface level is really "zigs stdlib has terrible and inconsistent short names similar to libc and posix", which isn't really a good measure of anything in particular. It's perfectly functional along this axis and par for the course at the systems programming level.

The toolchain approach has a lot more of a "hacking for hackers" feel, so when you hit bugs and so on it's very typical for folks to be patching their stdlib locally for a while or building their own toolchain to overcome the problems. I spent a little bit of time there recently with building ghostty on windows and was regularly messing with my toolchain and stdlib to make forward progress. Along these lines it's also much less complete - which is largely a function of being much newer, but you can take a ton of rust projects today, particularly things in the GUI space and build them for every major target platform with nearly zero effort. Zig is very very far off that, and there's going to be a need for a lot better platform level abstractions to get there. Rust did a great job with platform abstractions, which sadly was best documented in an anti-go inflammatory articles, but the point stands and if more generalized stands against zig at times too (https://fasterthanli.me/articles/lies-we-tell-ourselves-to-k... and other similar rants he wrote), though not all points port over.

The LLVM removal kind of move is somewhat enabled by this looser approach, which is also helped by the kinds of users the language has, and the smaller ecosystem. Another way of putting that might be "this is the right time to do this", as doing it later might lead to far more user pain and community noise or negativity. It's great for the world as we need more diversity, but it's also not all roses. At my current work we tried out Zig for hermetic cross-builds something that a lot of people tout as a strength. What we found was that the intrinsics that were written in pure Zig were sufficiently far behind libgcc/compiler-rt that it did not have sufficient performance for our use case - literally the binary couldn't handle our production load. Again, this is the kind of thing that can and likely will improve with time, hell if it was a priority I would have done it, but we had other solutions. Point is it's not as simple as a "this vs that" outcome, these moves have long running implications that may or may not affect a particular target - as an example it didn't really harm ghostty at all.

When you talk about culture each of these ecosystems has it's own dominant culture and a wide set of subcultures. How you choose to integrate with those, if you do at all, is up to you. Some might be more attractive for sure, and some might provide a different risk profile for different use cases as well.

Just off the cuff if I was scaling up a team for a professional project with a long lifecycle, I'd probably lean toward rust right now as it has a good balance of stable evolution and production readiness, without being anywhere near as stagnant as C++ despite much effort to move the needle. If I was in a really hacker mood where I just want to twiddle and mess with stuff, I'm not excessively performance sensitive (beyond the general order of magnitude that native compilation and near zero abstractions gets you), and my team is going to remain small and expert "everyone cracks open the source" folks, then I might pick Zig. These days I don't have many good reasons to pick C anymore. If it's patching a pre-existing thing there's no choice of course, but other than that it's mostly going to be "I'm throwing a 30 minute build onto arduino and don't wanna go off the beaten path for this project" kind of thing.