Item — HN Reader

Warning: link auto-plays video with sound.

Edit/addition:

AVX-512 in Skylake is pretty exciting. 32 FLOPS per clock cycle per core is amazing. Also good to see 64 or 128 MB of eDRAM being included in more configurations. It should also somewhat help with algorithms with larger working sets, not just with graphics. Integration is also interesting, latency between GPU and CPU should be low. This may open chances for better integration and sharing of work between those units.

1 Gbps ethernet starts to be a limitation, it's barely enough for current internet connections in some parts of the world. Nearly any SSD can do sustained 5 Gbps, most new mechanical disks push 2 Gbps. It's time for 10GBASE-T to be a standard, with power saving option to function at lower speeds. Nowadays even low end hardware such as $80 Asus etc. wifi-router with USB 3.0 can push 500 Mbps+ at file serving.

2 comments

+1 on the networking angle, although Intel does have Thunderbolt, which at least gets you 10Gb peer-to-peer networking if your hardware and OS support it.

It does seem very weird that 10Gb copper networking has existed since 2006, but hasn't fallen down to prices so low that it naturally replaces gigE. It seems entirely possible at this point, that 802.11ax will be finished before 10GBASE-T becomes widespread outside of server applications.

I as well desire 10GBase-T that is affordable. It seems to be very slow coming. I think the issue is that while there is some business desire for this, the consumer only wants wireless -- thus that is where all the effort is going. Probably to get past 1GB in the next few years, one will deploy a wireless network (although wireless suffers horrible congestion as opposed to switched networks.)

It is hard to get mainstream usage of CPU extensions until they are on the large majority of chips because creating multiple code paths is a pain. Thus while AVX-512 is nice, I do not expect to see usage of it outside video codecs until at least 3 years from now, and it likely won't be widely used until 5-7 years from now. This problem is compounded by the slow upgrade cycles of PCs now that their performance doesn't drastically increase per generation.

It's worse than that. It's still problematic to even assume SSE4.2, not to mention AVX2, which would help significantly. Virtualization is making the problem worse, because hypervisors set CPUID bits to lowest common denominator within a vMotion / teleportation domain.

I think (specialized) JITting is the answer for that. Generate code on the fly. I've been lately playing around with JITted [de]serializers, etc. I think it might be possible to have an order of magnitude gains in JSON, XML, msgpack, etc. processing.

Even *printf can be much, much faster. If the format specifier is static, first run just generates code just for that specifier. Subsequent invocations can be about 10-20x faster versus normal style format string scanner.

Edit: When I say JIT, I'm not talking about JVM or any other mainstream implementation specifically, but the concept in general. Which means just-in-time generation of (optimal) machine specific code.

> Even *printf can be much, much faster. If the format specifier is static, first run just generates code just for that specifier.

There's no reason you can't do what you want today AOT in stronger languages. Adding a JIT, or special AOT treatment of any given set of functions, to workaround weaknesses in C and its poor standard library, is just sad. Another problem is that stock libc printf has to be able to deal with localisation, and locales can be changed arbitrarily at runtime (although it's not thread-safe to do so)

One way to get all the performance out of modern hardware would be to stop using languages and libraries that haven't seen major improvement since the 90s, and still can't deliver optimal execution of functions, like printf, designed in the 60s.

Cough existing codebase cough.

We can significantly boost performance for some aspects of C/C++ in the meanwhile by JITting. "sprintf" is just slow. (side note: Also C++ "<<" stream implementations I've seen are unbelievably slow (about 3x slower than *printf) and tend to have a lot of side effects, such as a barrage of system calls if unbuffered -- the 99% case. Although I'm sure this is fixable with a better stream implementation and maybe some compiler assistance. Just try to benchmark stringstream...)

Otherwise I agree with you of course. Time is up for C/C++. Although I think they'll stay around. They're just not a good fit for modern CPUs anymore. There just isn't a good replacement yet. Maybe Rust will be that one day?

The problem with C isn't that it's old, unsafe, conservative, or that it lacks GC. The problem is that people have religious wars about strncat vs strlcat, which are both terrible and both equally broken, rather than looking at the big picture. The big picture is that there are hardly any functions in the standard library that aren't terrible.

If you wanted a safe, fast printf function in C for example, one way to do it would be to provide a function that took a format string and returned a reusable handle, much like you use regular expressions in a library like PCRE. This would give you instant benefits when it came to print and scan loops (no re-parsing of the format string), while still giving the compiler ample opportunity to do type checking without too much work. It even opens the door to a JIT and/or AOT backend, all without major changes to the actual compiler.

    void (*hello)(char const*, int);
    if (!printf_jit ("Hello %s, you have %d points\n", &hello)) {
        /* jit failed */
    }
    hello ("Steve", 80);
    printf_free (hello);

I don't lump C++ in the same boat. The above code can be written semantically in C++ today, without any external libraries, with full type-checking.

Nice idea how to get type checking. I was thinking more of a macro that initializes a function pointer with a code generator and subsequently overwrites it once JIT has been completed.

Sadly in both cases, no chance to inline or avoid unneeded stack use. Sadly the parameter string ("Steve") needs to be scanned for zero terminator -- this is slow. Zero terminated strings are an evil invention.

Is there any reason that it has to be just-in-time? Maybe there is potential for ahead-of-time SIMD optimization of precompiled binaries. There would be lots of tricky details, but perhaps an "sufficiently intelligent" installer could scan for SIMD optimizations and patch at install time.

Or maybe this could be built into the CPU cache? We already have a 1000-plus opcode decoded instruction cache. Already certain 'idioms' are optimized at decode time. How impossible would it be to have an opcode optimizer that rewrites the instructions in the cache to use the widest vector instructions available?

Currently CPU capabilities can even change on the fly, because of virtualization technologies like vMotion. That's why CPUID flags need to be masked by the hypervisors to hide CPU features that some CPUs in the same domain don't have.

Maybe that is possible. But would it be simpler? Maybe it can be a part of the solution?

Why would you care? You can generate a lot of code in a microsecond.

> Or maybe this could be built into the CPU cache? We already have a 1000-plus opcode decoded instruction cache. Already certain 'idioms' are optimized at decode time. How impossible would it be to have an opcode optimizer that rewrites the instructions in the cache to use the widest vector instructions available?

Sorry, but I don't understand your question.

Why would you care? You can generate a lot of code in a microsecond.

Because the limiting factor isn't code generation speed, but pattern recognition. Also, having spent too much time staring the mangled junk produced by optimizing compilers that concentrate on speed of compilation, I'm naively hoping that dedicating more processor time to optimizations will produce better code.

Sorry, but I don't understand your question.

CPU's no longer decode x64 instructions for tight loops. Instead, they inject decoded opcodes by that they index by starting address. Already, simple optimizations are being made at this level: https://sites.google.com/site/paulclaytonplace/andy-glew-s-c...

I'm wondering how feasible it would be to "rewrite" series of scalar instructions to a smaller number of SIMD alternatives either as part of the front end decoder or as something that continually optimizes the stream of opcodes in the cache.

> Because the limiting factor isn't code generation speed, but pattern recognition. Also, having spent too much time staring the mangled junk produced by optimizing compilers that concentrate on speed of compilation, I'm naively hoping that dedicating more processor time to optimizations will produce better code.

You wouldn't need to care of the actual generated code. It would be generated just and only for the currently running processor and memory subsystem.

> I'm wondering how feasible it would be to "rewrite" series of scalar instructions to a smaller number of SIMD alternatives either as part of the front end decoder or as something that continually optimizes the stream of opcodes in the cache.

Right. First you'd need to have available instruction encodings -- for x86, they're in very short supply. Secondly this would increase the amount of state that needs to be stored to memory and restored when context switching between threads and processes, unless the support would only be available to the operating system and opcodes are shared by all thread contexts.

You can achieve this by JITting already, without the context switching overhead.

On the positive side, this would ease instruction cache pressure.

I don't think we're talking about the same thing. I'm probably not helping by being sloppy distinguishing between opcodes and µops. But I'm referring to the hardware decoded µop cache that mostly replaced the "loop stream detector" in Intel processors post-Nehalem. No context switching, no OS support, rather hardware optimizations targeted at the interface between the outward facing x64 (CISC) instruction set and the post-encoding (RISC) µop driven monster within.

It would be a box along the same path as the one labelled "uOP Cache Hit Logic": http://pc.watch.impress.co.jp/video/pcw/docs/601/851/p16.pdf

I guess the other solution is to use OpenCL and write all kernels in that language. Then it should JIT into AVX-512 if it is available.

It might be acceptable for some subset of "embarrassingly parallel" problems. For other problems, writing OpenCL code that even approaches hardware potential is very hard. We might need other intermediary languages than it.

Currently LLVM is a better fit than OpenCL in general.

If you want the last 50% of performance, hand coding seems to be unfortunately the only way. Very little code needs to be that fast, though.

JIT and AOT on-device is the future, I agree. How good are the JVMs at taking advantage of new instructions?

When I talk about JIT, I think primarily specialized form of them.

I don't see much potential in JVM in its current form to be able to take advantage of this. Currently pattern detection and intrinsics ('fake' method invocation that is translated to just 1-3 instructions) are possible.

JVMs should especially offer better control over data memory layout. Number of references (pointers) required needs to be radically reduced. Objects should form continuous memory regions, and have a minimal number of references to elsewhere in memory. Inlining should be used as much as practical instead of pointer chasing. Unions would help too with vectorization, but would understandably significantly complicate the optimizer.

For example, an empty or a short String should be just 16 object-inlined bytes. Larger ones could still of course have references to byte or char arrays. UTF-16 should be removed in favor of UTF-8.

My understanding is that JVMs do not generate any advanced instructions from standard Java code. The only time they use these instructions is if they link in hand coded C++ libraries that make use of them.

Edit: Stackoverflow answer says that in rare cases the JVM can vectorize simple loops: http://stackoverflow.com/questions/10784951/do-any-jvms-jit-...

Just take a look at the code they generate, it's easy. In short, they do generate SIMD code, but very rarely take advantage of vectorized execution.

Why aren't the processor manufacturers making their own JITs and AOT compilers? Surely that would be a competitive advantage.

Intel makes a famous AOT compiler, ICC. https://software.intel.com/en-us/intel-compilers

> creating multiple code paths is a pain

GCC added some cool functionality to make this a lot easier recently:

https://gcc.gnu.org/wiki/FunctionMultiVersioning

No it isn't. You are still going to need to maintain multiple code path.

Especially with SIMD instruction set like AVX-512 -- to be performant you need to do things with direct assembly anyway.

Seems nice, but isn't all that much different from checking support yourself and initializing function pointers to correct functions - more work, but portable.