▲ 212 points
back
166 comments
I have written a lot of SIMD for both x86 and ARM over many years and many microarchitectures. Every abstraction, including autovectorization, is universally pretty poor outside of narrow cases because they don’t (and mostly can’t) capture what is possible with intrinsics and their rather extreme variation across microarchitectures. If I want good results, I have to write intrinsics. No library can optimally generate non-trivial SIMD code. Neither can the compiler. Portability just amplifies this gap.
I think a legitimate criticism is that it is unclear who std::simd is for. People that don’t use SIMD today are unlikely to use std::simd tomorrow. At the same time, this does nothing for people that use SIMD for serious work. Who is expected to use this?
The intrinsics are not difficult but you do have to learn how the hardware works. This is true even if you are using a library. A good software engineer should have a rough understanding of this regardless.
I think what a SIMD library does, above all else, is get the programmer to write code in a way that can be directly translated into SIMD instructions. A big issue that compilers have to contend with is that they aren't allowed (unless you enable ffast_math) to rearrange floating point operations. Putting an add or a multiply in the wrong place can spoil SIMD optimizations that the compiler could otherwise pull off.
But the problem is as you state. For people that really care about that sort of thing, they are likely going to have the exact SIMD sequence they want to execute in mind anyways. That leaves you with a definition that is doomed to be both not low level enough and too low level.
I think what this is useful for is a fallback description of the desired SIMD operations. It won't be ideal on non-targeted platforms, but it will be something.
For me the main issue is that if you're serious about SIMD, you need to use a state-of-the-art library and can't rely on some standard library whose quality is variable, unreliable, and which is by design always behind.
> I think a legitimate criticism is that it is unclear who std::simd is for.
I think it's for people like me, who recognize that depending on the dataset that a lot of performance is left on the table for some datasets when you don't take advantage of SIMD, but are not interested in becoming experts on intrinsics for a multitude of processor combinations.
Having a way to be able to say "flag bytes in this buffer matching one of these five characters, choose the appropriate stride for the actual CPU" and then "OR those flags together and do a popcount" (as I needed to do writing my own wc(1) as an exercise), and have that at least come close to optimal performance with intrinsics would be great.
Just like I'd rather use a ranged-for than to hand count an index vs. a size.
> People that don’t use SIMD today are unlikely to use std::simd tomorrow.
I mean, why not? That's exactly my use case. I don't use SIMD today as it's a PITA to do properly despite advancements in glibc and binutils to make it easier to load in CPU-specific codes. And it's a PITA to differentiate the utility of hundreds of different vpaddcfoolol instructions. But it is legitimately important for improving performance for many workloads, so I don't want to miss it where it will help.
And even gaining 60, 70% of the "optimal" SIMD still puts you much closer to highest performance that the alternative.
In the end I did end up having to write some direct SIMD intrinsics, I forget what issue I'd run into starting off with std::simd, but std::simd was what had made that problem seem approachable for the first time.
> If I want good results, I have to write intrinsics.
Any good book / article / tutorial to begin learning it? Most docs I've tried assume you already know what you can do and focus on how to express it in assembly or with intrinsics. However, I'm not even aware of what operations are commonly implemented in hardware. For example, FMA is not normally tought, is not expressed in higher-level languages... how am I supposed to imagine such a thing exists, even before looking it up in a reference manual?
In such discussions, whenever you mention abstractions are universally "pretty poor", to the extent anyone is listening, I think this hyperbole can do real damage. Maybe it prevents people from getting relevant performance gains, even if not 100% of the optimum, which is anyway unattainable. And what is the alternative? Not many projects can afford to hand write intrinsics for all platforms. And are you aware that Highway is basically a thin wrapper over intrinsics, which you can still drop down to where it helps?
Yep, same here and agree.
Compilers have definitely got better though: another issue in the past (maybe still is to a degree? although compilers have got a lot better at this in the past 15 years, but it used to be one of the things only Intel's ICC actually got right), that if you wrapped the base-level '__m128' or 'float32x4_t' in a struct/union in order to provide some abstraction, the compiler would often lose track of this when passing the struct/union through functions (either by value or const ref), and would often end up 'spilling' (not entirely the correct terminology in this context, but...) the variable from registers, and just producing asm which ended up uselessly loading the variable again from a stack address further up the call stack, when it didn't actually need to do that. So that was the situation even when using intrinsics within custom wrappers.
From 2011 to around 2013 ICC seemed to be the only compiler on amd64 which wouldn't do this. If you passed the actual '__m128' down the function call chain instead, clang and gcc would then do the right thing.
> I think a legitimate criticism is that it is unclear who std::simd is for. People that don’t use SIMD today are unlikely to use std::simd tomorrow. At the same time, this does nothing for people that use SIMD for serious work. Who is expected to use this?
There is plenty of vectorization that are simple enough to be done with std::simd today and that will still bring any autovectorizer begging on its knees for various reasons.
As an anecdote, I currently got a 8x speedup with std::simd (AVX2 & SVE2) on a rather trivial parser of mine recently that autovectorizer failed miserably to do properly.
Would I have get better result using intrinsics ? Likely, yes.
Did I want to suffer the maintainability and portability pain associated with it for a simple parser ? Certainly not.
For these use case, std.simd does the job. And will probably do a better and wider job with time when it get enriched by the committee.
The blog brings some valid criticism but really looks like a flame war trying to destroy an already opened door.
(1) Is there more performant solutions that std::simd for vectorization ?
Yes, of course. The STL evolves slow, its main goal is to provide a generic and portable implementations of a set of algorithms. Not to provide the best implementation in existence.
The best implementation of most algorithms (including SIMD patterns) evolves every 6 month, you can not expect a standard library with 3 different implementation to keep up with that.
(2) Is the future of vectorization ISPC ?
Nope. ISPC has been around for > 10y and is still niche. There is very good reasons to that: Yes it can generate better code but in most use case, adding a massive dependency of a compiler + an arbitrary LLVM version + a DSL on your project is not worth it.
Specially considering that it is an Intel project and that Intel (almost) abandonned the project multiple time (In pure Intel fashion).
So yes, criticism is easy, and yes std::simd is full of problems.
But I am glad it exists, and thanks to the people that made it happen... Because it is useful, even in the current state.
Do you say that from the perspective of compiled languages? I hear good things about .net core wrt SIMD, but that has the advantage it can decide at JIT.
Is this a technical impossibility or just it hasn't been done yet? Could a library support generating intrinsics for a large set of architectures?
Autovectorisation is the main way SIMD hardware gets put into use, whether you think it's pretty poor or not.
SIMD came to mainstream in 1995 Pentium MMX and has been proven rather difficult for compilers to target, but after 30+ years is doing a bit better despite PLT conspiring against it. (see eg CUDA, Futhark etc)
> People that don’t use SIMD today are unlikely to use std::simd tomorrow.
why? at least I see that I will start with std::simd in my pets. If this would not enough, I would go forward to intrinsics. But, I think, starting with std::simd would be much simpler for beginner.
> I think a legitimate criticism is that it is unclear who std::simd is for
It's for people that don't use SIMD today.
SIMD is hard, or at least nuanced and platform-dependant. To say that std::simd doesn't lower the learning curve is intellectually dishonest.
---
Despite the title, the primary criticism of the article is that the compilers' auto-vectorizers have improved better than the current shipped stdlib version.
what about Google highway project?
The point about the optimizer only seeing "opaque templates and function calls" makes little sense.
First off, templates are the opposite of opaque due to the fundamental requirement that the implementation be visible to every translation unit using a template. This makes any function calls trivially inlinable.
Second, and the reason for the above requirement, templates are compiled by monomorphization – making a distinct, separately optimizable copy of each concrete instantiation of a template. By the time the compiler backend sees the intermediate representation, there’s nothing about templates left.
There are of course reasons why highly abstracted template code may be difficult to optimize, for instance if function call chains are so deep that the inliner gives up. There are also legitimate reasons why a fully language-based solution might beat a library-based one. But one of the points of adding a library to the std is that the standard library is allowed to cheat as much as it wants. It can be deeply integrated to the compiler and implemented entirely using compiler magic if necessary.
std::simd may be too little, too late for many reasons, but I doubt any of them is that the compiler can’t see through the code.
As a compiler guy, the complaint about "opaque templates and function calls" to me raises serious doubts that the author has any idea what they're talking about. std::simd is designed to be akin to taking vector operations as intrinsics on <4 x f32> and similar types and wrapping them in a more C++ dialect than bare compiler intrinsics (and then a second layer on top of that to make things somewhat more portable).
So the implementation of all of the std::simd at the bottom should be tiny functions that map to essentially a single instruction, specified via a header file in a mechanism that guarantees you always have the body. This makes the functions trivially obvious candidates for inlining. Since it's a C++26 addition, the dispatching logic through the layers can largely be done via if constexpr, which means most of the code is discarded by the frontend.
Given that the complaint seems to be about not vectorizing a call to a sin function, it's possible that it's implemented in libstdc++ in such a way that the library doesn't know about the compiler's -fveclib implementation. But then again, the complaint is based on the libstdc++ v14 implementation of C++ Parallelism std::simd, not the v16.1 implementation of C++26 std::simd, which is completely different (and landed circa 2 months ago).
Compiler guy here - yes, the optimizer claim is,in fact, just totally wrong. The claim is it can't be constant folded, scheduled, etc because it doesn't turn into simd instructions. At the bottom, as you say, it will turn into code. That code is almost certainly builtins or uses the vector extensions or whatever , not raw asm statements. compilers can and do turn these into ir level simd instructions.
I agree with you, but just a small nit:
> First off, templates are the opposite of opaque due to the fundamental requirement that the implementation be visible to every translation unit using a template.
That's not strictly true, you can have an implementation hidden in a separate TU, as long as that TU instantiates the template for all template arguments that the users are going to use.
it's just a compiler hint just like all other hints, parity with other languages.
I made the first proposal to the C++ standard committee to introduce SIMD in 2011, before Matthias Kretz got involved with his own version (which is what became std::simd). This was based on what eventually became Eve (mentioned in the article).
Back then, it was rejected, for the same arguments that people are making today, such as not mapping to SVE well, having a separate way to express control flow etc.
There was a real alternative being considered at the time: integrating ISPC-like semantics natively in the language. Then that died out (I'm not sure why), and SIMD became trendy, so the committee was more open to doing something to show that they were keeping up with the times.
> There was a real alternative being considered at the time: integrating ISPC-like semantics natively in the language.
I think this is the best solution for truely portable SIMD. Sure it doesn't cover everything, but it makes autovec explicit, guaranteed and more powerfull.
One of the biggest problems with "portable" SIMD libraries, is that when it's used for simple things, often autovec is better, as it has access to the direct ISA semantics and can much easier do things like unrolling.
To me it’s clear adding the ability to express intent to parallelise is the Right Thing. This is the only way the compiler can actually know what you want it to do.
Trying to abstract over SVE with a SIMD library is a bit of a fool's errand. The intended programming model is just too different from traditional ISAs, and there are algorithms that are nearly impossible to write efficiently for it. All the ones I've seen wrap it up as a bastardized fixed length ISA, and even ARM's own guidance basically recommends that approach.
Frankly, the length agnostic stuff is a mistake that I hope hardware designers will eventually see the light on, like delay slots.
The linked[1] "six reasons to use std::simd" was just what I needed after a long week. Hilarious!
That certainly convinced me. When I was doing my taxes recently and had to watch those forced loading animations, I kept asking myself "why can't my compiler do this?" Thanks to std::simd, now it can!
isn't that just QoI issues? There's a reason why the libstdc++ folks labelled their implementation as experimental.
It should have been "eight reasons to use std::simd". Inefficient.
This is the first time I've seen a classic for loop called a "boomer loop", but apparently this isn't even the first instance (not the first definition):
"The Default Width Problem" -- this section seems confused and definitely reeks of LLM authorship. It's comparing -march=native against std::simd and complaining that std::simd<T,8> breaks portability with pre-Haswell. This is a real issue, but -march=native is no better! It bakes in the SIMD width at compile-time as well, so that binary also won't run on a pre-Haswell machine. It's a real issue but neither side solves it. You need runtime dispatch (a la Google Highway) to solve this.
GCC already solved it:
https://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html
The operations behave like C++ valarrays. Addition is defined as the addition of the corresponding elements of the operands. For example, in the code below, each of the 4 elements in a is added to the corresponding 4 elements in b and the resulting vector is stored in c.
One thing I will point out is that the code in the article is compiled with `-march=native` and `-ffast-math`, meaning that they're really only compiling for the exact same machine they are running on and no other. This seems like it is mainly applicable to places which can easily recompile code for the exact known hardware that they run on, such as HFT and some scientific computing.
Places which compile code to distribute for people to run on a variety of processors and platforms (or that require floating point code to be consistent between them), i.e. games and applications, will still be targetting a low end baseline architecture and therefore have a different outcome. I can say that in this space we are only now reaching the point where we can start compiling for AVX2, as we can expect the lowest end-user processor to support it.
Curious if people here have looked at the upcoming SIMD support in Go: https://go.dev/doc/go1.26#simd
Currently experimental, but looks like the first Intel arch will arrive in the next release in about 3 months. They are also going to support a portable layer.
Wondering what people here think about the approach the Go team is taking; I think they would appreciate more eyeballs on their design. (I’m not competent in this space (yet))…
Just write inline asm for x86 and aarch64 (if you care about that) and not care about the rest. Is it even useful to do simd on other processors?
Compiler optimizing even the code around the simd code based on the semantics of arithmetic or other things sounds silly after writing some of this kind of code
If you thought std::simd was a library nobody asked for, just wait until you hear about <linalg>. I feel like half the people looking forward to that think they're just going to get standard C++ bindings to LAPACK, when instead they're probably going to get an unoptimized, slapdash implementation of LAPACK written by people who aren't good at BLAS.
As for SIMD itself, designing a good SIMD library is difficult because there are several different SIMD approaches and some of them work poorly for certain use cases. For example, you can take an HPC-ish approach of "vectorize this loop" (à la #pragma omp simd) and have the compiler take care of a fairly mechanical transformation. Or you can take an opposite approach of treating a 128-bit SIMD vector as a fundamental data type in your language. Which approach is better depends on your use case.
Agreed, fixed with vectors needs to be a language feature, better compile times and would solve issues for most people.
Personally, I think that like Clang way to adding GLSL like vectors and semantics would've gone a long way. SVE might be an elegant design, but in reality there are probably a multiple factor of game and other 3d code being written that needs vectors compared to other fields, and there limited vector sizes aren't really a problem.
And honestly, considering the story of AVX512.. with 512 bit vectors being removed from mainstream by Intel, do we really really need longer ones despite it being from a "scalable design"?
> When Google needed portable SIMD for production image and video codecs, they built Highway — not std::simd.
Sure, they left the committee years ago. I am not trying to claim any sort of direct causality, but it sure seems like this is a case where Google's presence on the committee might have prevented shipping boondoggles like this. Modules is another case where I think Google's feedback might have been able to steer the ship in a better direction.
If we want to improve cross-platform SIMD, in my opinion we should start by supporting more operations in LLVM IR. Like vector expansion (currently we only have expandload), runtime-known shuffle vectors, pdep/pext operations.
Also, let's stop with the "vector length agnostic" types being the sole option for SVE extensions. I'd rather write an optimized routine for a 16-byte machine I'm targeting and be able to upgrade it in 5 years than have "agnostic" code that wants to pretend like it would work amazingly on all platforms, but the machine I optimized it for is theoretical. I'm fine with recompiling my code, I do it every day. If I have an algorithm that's truly vector length agnostic, I can make the vector length a constant in my code that can change based on the compile target.
The article's point in a nutshell:
> The problem is that std::simd in 2026 is the 2012 solution arriving after the world moved on. The committee spent a decade polishing a library-based approach while compilers solved the easy cases automatically and ISPC solved the hard cases with language-level support.
I find it interesting that the C++ committee would make that kind of mistake. Shouldn't they know better?
Something everyone is missing is that this is just feature parity with other langauges, this is not c++ specific. They added it because other languages have these compiler hints which are then (usually) used in llvm as opaque types for 'smarter' optimizations. Hand rolled code will still be better, but there are very niche instances where you want llvm to know this information, especially important when you don't care about performance, but you care about data integrity and obfuscation.
I'd actually rather just have the compiler give some guarantees on producing SIMD code when you write regular C++ code doing sums, multiplications, etc... in a particular way. And perhaps add a few more operators/keywords to the language for modern CPU instructions (we got things like popcount, countl_zero and fma, but what about e.g. pext, pdep, aes, ...)
sigh
C++ sits on that weird abstraction level where it wants to be a higher level language but it keeps grinding their gears on stuff like pointer sizes, pointer arithmetic or vector sizes and at the same time wants to keep being C compatible and needs that interface with the lower level world
Now compare with how numpy does things: you care about the data size but not the implementation.
Still, I didn't expect less (of a crap fest) from the C++ committee as presented here
Nobody should read that AI slop article. Nobody.
Maybe there's an interesting story in there, it's certainly possible. But the "author" could not be bothered to write it, and so why should we waster our time reading it?
Unnecessarily negative article. Lets not forget how awful C++98 was for years. Standardisation doesn't mean useful.
> The pattern is clear: every major project that actually needs portable SIMD in production chose a third-party library or a different language
of course they chose third-party library, because C++26 is just only published and don't have wide support/adoption experience.
> And the most damning data point might be EVE itself — a committee member looked at std::simd, decided it wasn’t good enough, and built his own library.
It's just a manipulation. First commit in eve[1] was published in 2018. There was no any std::simd in standard at that time.
> Nobody waited for std::simd. By the time it ships in C++26, these libraries will have a decade of production battle-testing, real user feedback, and cross-platform coverage that std::simd can’t match on day one.
Manipulation 2. No library have decade "of production battle-testing, real user feedback, and cross-platform coverage" on day one. So, why their authors created them?
> Including <experimental/simd> pulls in deeply nested template machinery — simd.h, simd_x86.h, simd_builtin.h, and friends. A trivial function computing sin on a SIMD vector takes about 2.2 seconds to compile. The equivalent scalar for-loop? 0.2 seconds.
Would be more interesting if you compare this with precompiled headers and C++20 modules.
> The std::simd version? It emits actual vsqrtps + vmulps because the optimizer can’t perform algebraic simplification through opaque template function calls:
opaque template function calls? What is this?
Of course there is 1000 examples when compiler can do better job with scalar loop. And there is 1000 examples when it can't. But, for some reason people do write simd manually. Probably because they want predictable code generation - no massive slowdown on another compiler/another compiler version/another cpu/another one line of loop changed.
> sqrt(x) * sqrt(x)
what compiler would generate with manual simd intrinsincs? I doubt the same as scalar mul.
> The frustrating part is that the problems are well-understood. SIMD programmers have been asking for the same things for years, and none of them are in std::simd.
Show me your proposal with critique of std::simd, if you asking for them. Or at least someone other proposal. How you can understand that someone asking?
Too many emotional statements in, too little technical details.
Why did you use an LLM to write this?
Why not just writing inline assembly is not enough?
You optimize for a specific target.
The problem is that you cannot be cross-platform. Sure.
But that is why software is incremental.
I write for my HW, not yours. You can write for yours.
Make folders with implemntations
x86_v1 x86_v2 arm64 riscv64 ... ... ...
and include
Slop.