back
49 comments
why did CPython running the Mandelbrot benchmark have a running time of 134.419s and an incredibly wide confidence interval of ± 32.347s? I had no idea, so, in the best tradition of VM measurements, I buried my head in my sand and hoped that no-one would say anything. They didn’t.

A beautiful summary of the field!

In other words, this VM is highly optimised for this benchmark — but no-one ever ran it long enough before to notice that performance gets much worse over time.

Perhaps, although more likely, they noticed; and then in accordance with tradition, promptly buried their heads in the sands.

---

Since this thread is likely to be read by people who care about unexplainable performance glitches, I'll highlight an otherwise inexplicable ~20% run-to-run performance drop that I recently saw an explanation for: https://www.realworldtech.com/forum/?threadid=179700&curpost...

There has been a lot of talk about whether AVX512 is of real world benefit. One big issue is that it's so power hungry that the entire core compensates by slowing down whenever 512-bit instructions are executed. What's not well known is that with an AVX512 capable processor running the common Ubuntu 16.04, you may well encounter this slowdown even if your program never uses AVX512.

More specifically, there is a problem with the way that some versions of glibc are saving and restoring the 512-bit ZMM registers. Even at program start-up, the upper 256-bits can be in a "dirty" state, which can cause the core to slow down to the "AVX512-light" speed on any SSE, AVX, AVX2, or FP operation. Patches for glibc were backported as far at 2.25, but Ubuntu hasn't included these fixes in their updates.

The bottom line is that if you are working on an AVX512 capable system, you should probably make sure that you understand this bug before publishing any benchmark results.

Just for completeness, this problem with AVX is biting even when using containers instead of VMs.
Just for completeness, the "VM"s in this article have nothing to do with containers. But of course we all read the article before commenting and we all know that, right?

Without being an expert in virtualization or containerization, it seems pretty obvious to me that if a certain instruction causes thermal throttling, then that'll cause thermal throttling whether the instruction is issued from a host OS or a guest OS or for any other reason. Indeed, the more layers of abstraction involved, the likelier you are to hit the problem.

One thing I am struggling to understand is why do we need JIT in the first place? What would be wrong with statically compiling java or c#? Portability is nice but the reality is that almost no one uses it. Typically you have to create different packages for different platforms anyway just because of the installer. Most backend applications are designed to work on a single platform (who is running a mixed windows+linux/x86+ARM backend for the same binaries?). For front end applications, most platforms have different UIs, and all new platforms (android, iOS) have incompatible APIs anyway.

Is there a fundamental benefit I am missing?

> Portability is nice but the reality is that almost no one uses it. Typically you have to create different packages for different platforms anyway just because of the installer.

Strange you think that, because mobile phones use a variety of processors and architectures, and virtually all the binaries running on them were shipped with a platform-agnostic bytecode. Not to mention all the Java servers out there, or cloud services like Azure and Google Cloud which host .NET VMs.

As for your question, JIT has a few advantages beyond portability:

1. The fact that the bytecode isn't tied to the machine means the CPU and architecture has more flexibility to change, introduce new registers, vector units, and so on, and the VM can take advantage of these features without needing recompilation and redistribution of every program.

2. The machinery permitting runtime compilation is typically used for other things by programmers, like manually dynamically compiling new program fragments based on user input which are more performance-optimal (think LINQ expression trees and other forms of quasiquotation/explicit staging).

3. Programs can benefit from progress in optimization algorithms without needing to statically recompile and redistribute every program.

4. Bytecode is typically much safer and easier to analyze for various safety and security properties, so they can be shipped to clients on-demand for special purposes. Like web assembly.

As you can see, the main benefits aren't tied to performance, but to portability, safety and flexibility. I think the usual focus on performance is a red herring.

> As you can see, the main benefits aren't tied to performance, but to portability, safety and flexibility. I think the usual focus on performance is a red herring.

I'd recharacterize the performance question - the question as I've observed it has been "can we get all of these nice things without sacrificing too much performance", where the bar for "too much" has continually moved.

From the 100x slower days of early TCL implementations, to 10x, now down to some small percentage overhead.

It's good that the bar keeps moving - it puts pressure on the JIT world to keep delivering. I don't have the time to write up a long post about my personal thoughts on the long-term future of JITs, but I can summarize it thus (I realize it's a bit of polemic perspective):

1. I believe JIT compilation will slowly become the norm for most program execution environments - even for static languages. Wasm is a kick in that direction, and more will follow.

2. One reason I'm excited about JITs slowly making their way into static languages is things like proper first-class, _highly performant_ implementations of higher-order parametric polymorphism. For a quick example, I am always slightly annoyed when Rust tells me that I can't take a reference to a trait because it's not an Object trait:

``` trait Foo { fn bar<A>(...); }

let foo: &Foo = ...; // Invalid. Foo is not an object trait. ```

The reason we don't get to have this very nice, wonderful, powerful thing is because we need to specialize ALL occurrences of `bar`. I want this feature in my life. I've butted my head too many times against it and had to implement a compromised solution with type erasure and re-discovery on the other end of an API, leading to lots of unnecessary complexity, code duplication, and surface area for bugs.

A VM carrying a JIT that knows how to compile new specializations on demand effectively renders the notion of `object traits` obsolete.

Now Rust is not the right language for that (it wants to be bare metal and runtime-free and low-level). However, I would love for higher-level statically typed languages to start offering me these capabilities.

3. I actually believe performance with JITs will eventually be better than pre-compiled code. Our techniques are still young and somewhat ad-hoc. There are concrete areas where there is significant room for improvement in the design and implementation of JIT engines. This industry is still basically in its infancy.

You should spend some time reading about mainframe execution environments for z/OS and IBM i.

JIT has been the norm since they exist, code is compiled to bytecode and AOT compiled at installation time or on demand if the environment is changed in some way.

In a way, similar to deployment scenarios adopted by Android, Windows Store and to a smaller extent Apple devices.

And chips on smart cards, electricity meters, telephone PBX,...
The theoretic fundamental benefit is you are able to optimize for code paths practically occurring during the runtime with particular set of data or inputs. There is no way to know that at compile time, hence it's often touted as a way to make things "faster than C" with sufficiently smart JIT compiler.

But of course no JIT compiler ever is sufficiently smart and any execution path benefit gained is lost to poor cache discipline of VMs. And the whole JIT/hotspot concept was really a hindsight attempt to get away from abysmal performance of VM based languages.

You're forgetting that the whole idea is not to compile the same language better, but to compile different languages (that are supposed to offer better productivity) to have similar performance to C. Obviously, languages such as Javascript or Ruby cannot possibly be compiled AOT with good performance, but even Java relies heavily on virtual dispatch, and a JIT can inline through virtual calls. Languages that don't rely on virtual dispatch as much will benefit less from a JIT.

Today, as we have both AOT and JIT compilers for Java, we can see that the JIT does, indeed, generate better code. By how much depends on your specific use-case.

> lost to poor cache discipline of VMs

What are you referring to? If you're talking about memory layout, then this has nothing to do with the VM and everything to do with the language design. The reason Java has little control over memory layout was that it didn't make much of a difference in the '90s; now it does, so Java is getting control of layout.

But you don't need JIT for that. You need an intermediary language (ByteCode / MSIL). In a non-JIT world you would use ByteCode/MSIL libraries, and just before deploying, statically compile them for the platform they will be deployed to. JIT just means you are doing that step everytime the program runs (+/- caching) for every machine.
You cannot efficiently compile ahead of time languages whose semantics heavily rely on dynamic dispatch. AOT profile-guided optimization would help, but it carries added effort (and still cannot be as aggressive if there is no possibility of deoptimization and recompilation by a JIT). Intermediate representation is completely irrelevant.

Java now offers "deploy-time" AOT compilation, and the results are not as good as a JIT (YMMV).

Java now offers "deploy-time" AOT compilation for free.

It has been available since the early days in commercial JDKs and initially it was the deployment option on IBM mainframes, by converting JVM bytecodes into the bytecodes of the mainframe execution environments.

There is another very practical benefit: You can use all features available on your PC.

E.g. when you compile an application C-style, you set the target architecture to the lowest CPU you want to support. If this is , say, SSE, then no SSE2 instructions will be used by the program and the sometimes huge performance gains from them are lost.

A VM like hotspot can generate code for this specific machine. If you have SSE2, it will generate SSE2 instructions. If you dont, it wont.

Recently, I see more applications choosing code paths based on CPUID, but this only works for CPUs existing at compile time. If you app was compiled in 2016, in wont use CPU features from 2018. If your java app from 2016 runs on a JRE from 2018, it will use them.

I would really like to see a Java program beating a C program in this use case. Say, outperforming it (or even matching to same order of magnitude) on DCT/IDCT even by using SSE2 vs SSE/MMX. Can only say am immensely sceptical.

Another thing is these kind of gains with instruction level efficiencies are exceedingly rare and your typical VM program never runs any code where SIMD extensions would make a difference.

This article is not what you ask, but it is an example of a VM beating native:

https://nullprogram.com/blog/2018/05/27/

Java uses the same technique.

Basically, a JIT allows you to skip the overhead of the GOT.

Devirtualization is another case.

I have personally seen Java beating an existing C++ program for the reason I describe, but I cant publish the results here. You have to program carefully or you loose all speed benefits because of pointer chasing. But it is possible.

> Basically, a JIT allows you to skip the overhead of the GOT.

This is a problem that is GNU/Linux-specific. Windows DLLs are not PIC (position-independent code), so they do not suffer from this problem and don't need a GOT.

See

> https://www.symantec.com/connect/articles/dynamic-linking-li...

for details.

The method that Windows uses is much faster in the average case, but slower in the "bad" case. So GNU/Linux vs Windows have different performance tradeoffs.

But don't static compilers generate checks for the availability of these features and ship both versions? I guess it is rather to take advantage of new features that were not available when the code was compiled.
In other words "an obfuscated Gentoo".
"There is no way to know that at compile time"

Profile-guided, whole-program optimization can do that for whatever execution patterns the developer feeds them. It will be faster than the JIT's because it has more time to analyze and optimize. The JIT will be better on the exceptional cases the AOT compiler didn't optimize for. In many apps, I'd much rather have the AOT approach than the JIT approach. Heck, some routines in fast path might even get superoptimized if developer wants. :)

.Net relies on runtime code generation for generics. This allows .Net to support generic instantiation with a type parameter the compiler hasn't seen at compile-time. Thus I don't need the source code for your generic data structure to use it with my custom types. I can just link in the compiled library.

It also allows the .Net regular expression library to be able to dynamically generate IL-code to match a given regular expression, which is then dynamically loaded into the process and JIT compiled to machine code. Other projects surely use the same facilities to dynamically generate and run specialized code based on user input and/or environment.

Not when doing AOT compilation.
Indeed. The question was why do JIT instead of AOT, and I tried to list some benefits.
Ah ok, sorry for the pedantic comment then.
No worries! I realize I could have been more explicit in my original comment.
> Portability is nice but the reality is that almost no one uses it

That's not true — Everybody actually uses it in practice. The .jar you grab from Maven Central just works unconditionally, your CI server builds one single artefact that runs on your Mac and on the Linux server.

A JIT compiler can optimize based on the runtime behaviour of the code. For example, it can reduce the costs of runtime polymorphism by optimizing virtual method calls as if they were statically bound, and then when a second implementation shows up, deoptimize the code to make it work correctly. Here's an in-dept article about that: https://shipilev.net/blog/2015/black-magic-method-dispatch/
It's not just about portability, but also things like dynamic loading and runtime modification of code, which modern systems rely on quite a lot under the hood.

People often aren't aware of this because the actual mechanisms tend to be hidden inside libraries or tools they depend on.

> dynamic loading and runtime modification of code, which modern systems rely on quite a lot under the hood

Can you provide any example? I can't think of any and it sounds very interesting since you seem to be implying that it's common. I assume that "dynamic loading" means something more sophisticated than loading a dynamic library at run-time?

Dynamically generated code, e.g. proxy classes and aspect pointcuts being inserted into code and then recompiled by the JIT.

And even "just" dynamically loading a library requires recompilation because it will invalidate optimizer assumptions such as devirtualization (part of class hierarchy analysis in hotspot).

Portability is a great feature for us. Our servers are Linux, our desktops are Windows. I can start up the entire Java stack on my PC if I need to debug something. The C++ components have to run on the Linux boxes.

Of course, I could run the applications on the Linux servers and remote debug, but the round trip time for rebooting and getting back to the same breakpoint is longer with more manual interaction.

> Portability is nice but the reality is that almost no one uses it.

Um, no. The reality is that almost everyone uses it. It's extremely common for developers to develop on Windows or Mac and deploy on Linux.

> Portability is nice but the reality is that almost no one uses it.

On the contrary - for example, lots of people develop on MacOS and deploy to Linux.

Am I reading that right? There is a whole article about how long JIT compilers take to settle, but the actual performance changes by just a few percent tops? (for example, first graph has 0.571 for cold JIT, and 0.568 for warm JIT, a 0.5% difference)

I think for me, the main takeaway from article is startup time for JITs is irrelevant, since it does not affect performance much.

In general, yes, you're reading the timings right, but there's a couple of factors to consider.

First, the JIT compiler has often done most of its work during the first in-process iteration, so the plots are not generally following a "one in-process iteration is slow, then the JIT kicks in from in-process iteration 2 onwards" model. Put another way, the JIT compiler is often making even the first in-process iteration run pretty fast.

Second, I would suggest that the often small timing differences are more worrying than they may first appear. In a semi-mature compiler, optimisations are frequently in the range of a 0.5-1% improvement. So if your measurements are only accurate to (say) 2%, most of your attempted optimisations will be misclassified (bad optimisations will sometimes be measured as good; good optimisations will sometimes be measured as bad). Furthermore, when a VM recompiles things, it generally performs several optimisations at once. Thus, if the overall performance gets worse (even if by a small bit), it may suggest that several optimisations performed badly at the same time.

But doesn't that mean the article's point is kind of moot? That when people blame slow languages / VMs, they actually are slow and it has nothing to do with warm-up time?
Both can be true, benchmarking JITs is really hard, but even so, (despite decades of claims to the contrary), managed VMs actually are a bit slower than unsafe native code.
Nothing prevents native code from being safe
That was my reading of the graphs as well, and I was surprised it wasn't addressed in the text.
Interesting stuff. To learn that it can take 190 iterations for warm-up to finish may be a revelation to VM developers.

The idea about some runs having slowdowns might not be significant if it's just one run out of 30 that does it though. The result in the last Diagram of TruffleRuby on spectralnorm having 25 slowdowns and 5 no steady-state is interesting.

Full paper at : http://soft-dev.org/pubs/html/barrett_bolz-tereick_killick_m...

It would be interesting to see how much wall clock time these applications run for. One answer to the question of how many iterations to run for is to compare it to the expected running time of real applications. If the expected running time is minutes, and the VM hasn't warmed up after ~1 minute, then it's pretty irrelevant whether it would "eventually" warm up or not. This would be one way to side step the halting-problem-esque question they've gotten themselves into with respect to how long to run the benchmarks for (which I'm pretty sure is intractable).
Is anyone able to link to the krun software used?

What do people use to make their computers stable for benchmarking?

The best place to start with Krun is its GitHub page https://github.com/softdevteam/krun/
How do I make LaTeX tables like that?!
Using booktabs for a start. There are various other packages for making tables nice but that one gets you a long way. Also study well typeset tables to learn why they look nice.
>study well typeset tables to learn why they look nice.

Can you link to any resources that discuss this?

I don't know any. What I meant was just to find good looking tables yourself and study them with an eye for typesetting. The manuals for LaTeX packages, including booktabs, are often very good too.