Looking at it from an attacker’s point of view, a VLA is a primitive for adding an arbitrary offset to the stack pointer: extremely powerful and dangerous.
You can’t use VLAs for large allocations because the stack has a limited amount of space and you don’t know what that limit is. So you must check the size is reasonable before declaring a VLA. So you might as well declare a fixed-size array that is large enough for all reasonable uses.
Yes, one should not let an attacker control the size of the VLA. But in any case, one should use -fstack-clash-protection .
With stack clash protection if an attacker can control the size of VLA and it becomes to large, you can a trap with is likely DoS. With a fixed size array which you overflow instead, it is more likely a RCE. If you check the size, it does not matter.
That you can't use the stack for large allocations has nothing to with VLAs and also depends on what you call large and how large your stack is.
* automatic VLA (i.e. stack allocated ones) will stay optional
* VM types (e.g. VLA in function parameters) will be mandatory feature again
I mean, I'm yet to see a definitive example where automatic* VLA was actually right tool for the job.
I know they were supposedly introduced for the numerical analysis, but I fail to see what problem it actually did solve. Yes, the syntax way neater than piecemeal allocation, but it could have been solved with functions and macros anyway. Did VLA just hit some sweet spot in performance between fixed size arrays and heap-allocated ones, which made number crunching better optimized?
* as opposed to VM types used in function parameters and for allocating multi-dimensional arrays on heap
In terms of security, there are some issues but it is largely overblown and misunderstood. A dynamic buffer is always a sensitive piece of code when dealing with data from the network. VLA were the language tool of choice and then got a bad reputation because they were involved in many CVEs. But correlation is not causation. The main real issue is that - if the attacker can control the size of the VLA -, it is possible to overflow the stack into the heap. This can be avoided by using stack clash protection which compilers support only since a couple of years. With stack protection I believe VLAs can be safer than fixed size arrays due to improved bounds checking.
I think the main gripe most programmers have with VLA is lack of control over them. Fixed size array is, nomen omen, fixed and predictable - can even be tested beforehand. malloc() at least returns NULL on fail (although memory overcommitment muddies the situation), so program can take some action. But what happens when VLA fails? Segfault? Stack crash protection if compiled with it or worse if without? None of those options is graceful from the perspective of end user.
> This can be avoided by using stack clash protection which compilers support only since a couple of years.
As you yourself say, it's been only few years since this protection made its way into compilers. But that's not the issue. The issue is that `-fstack-clash-protection` isn't part of C language, it's part of compiler. What's the incentive for the developer to use less certain feature when there are easier alternatives?
The people at WG14, Google and Microsoft (VLAs are one of few C99 features not ever coming to MSVC) think otherwise.
So you're basically saying they are wrong.
Actually, they allow for some limited size checking in function parameters [0], but GCC added this feature only very recently
[0]: [link redacted]
https://godbolt.org/z/q9qsax7qY
(also compiler support is still improving)
It basically shows that if you use automatic VLA you have can do some checks on VLA.
But if I prohibit automatic VLA altogether and use fixed size array, I don't need to worry about that at all and the example falls back to what I presented with UBSan doing just its regular thing.