back
57 comments
The article says that in data center workloads, 15-20% of CPU cycles are spent in the kernel. Contrast that with Brendan Gregg's statement (for Netflix running on EC2): "Most microservices we have run hot in user-level applications (think 99% user time), not the kernel, so it's difficult to find large gains from the OS or kernel." [1] I wonder why the OP and Brendan had such different observations.

[1]: https://news.ycombinator.com/item?id=13081465

Right. Most cloud systems we have are user heavy, there's not much kernel, usually <3%. They cited a Google paper with 15-20% kernel, which is a lot. Although I've heard from other sources that Google is kernel heavy, so this further confirms that. Different workloads.

It'd be interesting if someone like sysdig would share (if they could) the usr/sys breakdown for their entire customer base (I seem to recall they were sharing other such statistics).

It's a bit hard to see how much an exokernel or unikernel would help given, say, a workload that was 5% kernel. You might say it would only improve performance by less than 5%. But that's just considering the cycles saved. Such kernels should also be smaller, and consume the CPU hardware caches less, leaving more room for the application to execute on those precious resources. That would improve the performance gain further than just the reduction in cycles alone.

Security and boot times are also relevant, no?

Unikernels are much easier to manage than containers.

Sure, both are relevant. Cutting boot time only helps so far: we also have to tune application warmup to be faster as well. This is a little dated (we've tuned it down since then), but here's a microservice with a 10 minute application warmup time: http://www.brendangregg.com/blog/2016-09-28/java-warmup.html
Could reduced MMU overhead be significant on a server?

I work in the embedded space, so very different with small CPUs. Using an MMU compared to a simple MPU can waste 25 to 30% of a CPU cycles easily. I wouldn't expect such overhead on a big server CPU with much bigger TLB (but also bigger working sets?), still a unikernel could use a very simple mapping with in some case a single large page, and see very little MMU management overhead. I have no idea on the possible gain on a server, but I'm curious. Any hindsight welcome.

25-30% is highly unusual.

There's something extraordinarily inefficient going on in your system.

This was on an ARM926 running a quite large code base. The ARM926 TLB has only 64 entries, which is not much when running an OS with 4 kB pages. So that's quite an extreme (and old...) case, but interesting when dealing with rather small CPU with not so small context. The MMU has a performance cost to keep in mind.

A server is very different. Still, a blog on transparent huge pages on Linux [1] shows an example where a high load JVM application server spends over 10% of its time doing page walk: "Yes, you see it right! More than 10% of CPU cycles were spent doing the page table walking.". So there's a massively bigger TLB, but also the server run much larger application with a large footprint. In the end, for some apps the MMU overhead can be significant in the server space.

In a unikernel, there's a single address space used and no isolation. You could map it with only huge pages, and dramatically reduce TLB misses even for a large footprint application (assuming a server running mostly such unikernels, as another regular OS apps could trash the TLB otherwise I guess). Now, I'm not sure there's any such large footprint application running an a unikernel yet, but it may be a possible gain for unikernels on principle.

[1] https://alexandrnikitin.github.io/blog/transparent-hugepages...

I wonder how this would compare to Netflix's CDN machines. Where Netflix moved TLS encryption into the kernel.
CDNs are kernel heavy. :)
Doesn't Brendan Gregg work on things like Illumos/SmartOS? I think it's probably using lightweight OSs like that which help

Meanwhile most people just kinda take out of the box Ubuntu

I've primarily worked on Linux for 3+ years.
In any case, I wouldn't consider illumos much more lightweight as Linux or bsd.. It's Solaris at its heart.. (not to disrespect Solaris, it is/was a great OS)
Great question. I'd guess that Google's workload differs from Netflix.

What's your guess?

Funny thing is, my old OS project [1] is more or less that - it could be a typical OS with separate tasks and so on, but it is effectively a single program, and can easily be compiled as a single program with all the standard calls replaced with OS-level implementations.

In practice, these are two different sets of use cases - in one you want clean abstractions, process separation, storing message passing and so on, in the other you want to get the call from your code into the libservice ASAP. Trying to (unwittingly) combine the two gave me quite a bit of grief.

Another disadvantage is a lack of any protection (at least in my implementation), since everything shares the address space.

Anyway, it's odd to see that a random idea i had 17 years ago is actually starting to get used.

[1] http://orbides.org/aprom.php

Yours sounds like a Language-based Operating System, except you didn't implement protection in the language. Anyway they've been around for a very long time. The Burroughs system that I'm familiar with was around in the 1960s (from 1961 according to Wikipedia).

https://en.wikipedia.org/wiki/Language-based_system

Isn't that basically what SQLOS is doing for Microsoft SQL Server[1]?

[1] cf. https://blogs.msdn.microsoft.com/sqlosteam/2010/06/23/sqlos-...

Back to real mode we go.
I guess we should have been listening to Terry all of this time!
I really wish I could do a good bit of my work on his OS. I'd much rather a Kawasaki than an aircraft carrier that is half sinking.

Charles Moore of Forth fame has an amazing interview where he argues that the OS is really not necessary on modern hardware. Of course this man built his own language, CAD software, and chips...some of which are in outer space. Impressive man.

I don't suppose you have a link to the Chuck Moore interview; I would love to take a look at it.

(Alan Kay and co. also argued that point.)

http://www.ultratechnology.com/moore4th.htm

https://www.red-gate.com/simple-talk/opinion/geek-of-the-wee...

The first one has where he mentions the OS. Personally, I do think the OS is needed, but not in its present form. Instead, something far far simpler than even Linux. Yes, I've seen Alan Kay's similar presentation on the amount of code in windows and the F-22 raptor. Stacked it is supposedly unbelievably high like many stories, while their Smalltalk Machine built at Xerox had the entire language, GUI, text editor, image programs...etc in a tiny amount of code. In the modern day I think of Rebol which has a very powerful and simple DSL for so many tasks such as creating GUIs and the full download is like a few MB with zero install. Truly a marvelous piece of software.

> the entire language, GUI, text editor, image programs...etc in a tiny amount of code

But it wasn't internationalized, accessible to people with various disabilities (e.g. blind people via a screen reader), automatically installing updates over the network, encrypting all communication over the network (as in TLS today), etc.

No doubt some bloat comes from accumulated incidental complexity. But to some extent (I don't know how much), one person's bloat is another person's essential features. For example, if you tell me about some bloat-free GUI toolkit that draws its own widgets (as opposed to a wrapper over multiple platform native toolkits), my immediate reaction will be "What about accessibility?".

[missing features]

Well, MS Office, for example, is around 20000x the code of early Smalltalk "Personal Computing" systems. Let's assume that that core code covers only 1% of the features, so you'd need 100x more code to implement all of them. I think this is being very generous, but let's stick with it.

That would mean you still have a factor 200 unaccounted for. In other words, even when being very generous, we still haven't accounted for 99.5% of the code.

kOS from http://kparc.com boots on metal, has a workable GUI, network, file system, text editor, database, programming language and a few other bells and whistles, in less than 500KB uncompressed.

No, it doesn't have accessibility or internationalization, but I suspect if it did (by the same people) it would not be much larger beyond the inherent complexity - e.g. Unicode directionality and glyph data; or sound waveforms for text to speech.

Most of the size of modern software, OS included, is accidental

> Instead, something far far simpler than even Linux.

Who said that Linux is simple?

Good point. I guess in comparison to Windows.
Where is Chuck Moore now? He used to write a blog that was sometimes quite educational.
https://groups.google.com/forum/#!original/comp.lang.forth/L...

Per this, he's retired now.

I believe he's still with GreenArrays.
He retired a week ago apparently.
Chuck and Terry's work prove that most kernels are just really terrible compiler systems
Terry A. Davis' TempleOS does not run in real mode, but in 64 bit mode (long mode) in ring 0.
I think what he really meant was real >time<.
> I think what he really meant was real >time<.

On "normal" x86 processors (say: the CPU that will be inside your PC/laptop if you buy one) it is really hard (I don't want to claim "impossible", but at least really hard) to do real-time stuff. Intel knows that and this is also among the reasons why they released their Intel Quark SoC/micro controller, which is perfectly suitable for real-time stuff.

Why is this the case? One obvious reason is that the caches (D$, I$, micro op cache etc.) und the pipeline stages (EDIT: including out-of-order execution) make it really hard to predict/prove/test strict upper bounds on the performance of some code fragment on "normal" x86 processors.

Another also well-known "issue" is that many modern x86 clock up or down depending on core temperate (I think this behavior can at least partly controlled by the firmware (UEFI)).

But there is a much more subtle reason, too: It is SMM (system management mode): https://en.wikipedia.org/w/index.php?title=System_Management...

For backward compatibility a lot of legacy functions are implemented via SMM calls such as emulating a PS/2 mouse/keyboard (while really a USB mouse/keyboard is connected). As long as you cannot guarantee that the code of your OS will not trigger such a function that is emulated via SMM, it is very hard to ensure real-time guarantees.

EDIT: To quote directly from the Wikipedia article: "Operations in SMM take CPU time away from the applications, operating system kernel and hypervisor, with the effects magnified for multicore processors since each SMI causes all cores to switch modes. There is also some overhead involved with switching in and out of SMM, since the CPU state must be stored to memory (SMRAM) and any write-back caches must be flushed. This can destroy real-time behavior and cause clock ticks to get lost. The Windows and Linux kernels define an ‘SMI Timeout’ setting a period within which SMM handlers must return control to the operating system or it will ‘hang’ or ‘crash’.

The SMM may disrupt the behavior of real-time applications with constrained timing requirements."

> Another also well-known "issue" is that many modern x86 clock up or down depending on core temperate (I think this behavior can at least partly controlled by the firmware (UEFI)).

From what I've read, current Intel processors simply can't be set to run at a single constant frequency.

If I can offload work to the OS, great! I don't want to manage the virtual memory backing my mmap when the kernel is really good at it. It is not inherently bad when CPU time is spent in the kernel.

The point I'm taking away from this article is that some workloads (probably not mmap) trigger bottlenecks in the kernel, and would benefit if it were possible to replace those kernel components via app library.

That said, we do things like this all the time today. For example, I can provide my own memory allocation library instead of depending on the one provided by the operating system.

So I'm unconvinced there are many real world bottlenecks without a descent solution available today.

I wonder: Is the real point that leveraging custom implementations is hard and we could all benefit from a nice central repository and a reasonable dependency manager?

Well, having an entire copy of Linux in a container is a bit much.
This seems like a trade-off we're already okay with making.

    $ du -h /boot/vmlinuz-4.4.0-93-generic 
    6.8M    /boot/vmlinuz-4.4.0-93-generic
    $ du -h /usr/bin/docker
    12M     /usr/bin/docker
I think you're misunderstanding what GP means by "linux" on purpose.
And conveniently forgetting that docker is written in Go, so the binary also has the runtime and all used libraries baked in
I thought this was going to be about things like Dynix
I'm currently using https://askubuntu.com/questions/138356/how-do-i-get-a-live-u... and really like it