There's also a significant overhead for not being able to target exactly the CPU model and parameters for the call at compile time. Unless of course doing profile guided optimization just for that CPU model and use case.

I've been playing around with an idea for a long time to write a JIT for native code. I think it's possible to speed up most native code by JITting it. Sadly it would take much more time I can afford to spend. The principle is sound, though.

It'd be possible to eliminate a lot of computation at runtime. Remove a lot of branches, use CPU model specific instructions when a suitable pattern is detected, etc. Functions could be simplified to constraints, function calls could be dynamically inlined. Spilled stack variables could be allocated in registers. Calling conventions optimized into passing more parameters in the registers. On register starved 32-bit x86, registers for parameters with effectively constant values could be converted into instructions with immediate values, saving registers for actually changing data.

There's also no reason why you can't cache previously JITted code into a file.