▲ 84 points
back
55 comments
This does nothing for use-after-free or integer under/overflow, so “Making C SafeR by Extension” would be a better title.
It’s good to see people attempting this, though. I would think chances are good Microsoft will use it internally, and that already would be a win. Also, it’s a clang/LLVM extension. That makes it easier for third parties to start using it.
On the other hand, their choice to use Ptr<T> for familiarity with C++ requires users who want to keep their code compatible with C compilers that don’t support this a bit harder (it will require wrapping that in some macro, say _ptrTo(T)). That isn’t ideal for adoption in open source programs.
Re. syntax. I don't understand, why can't they just make _Ptr an attribute like _Atomic, and provide a header <stdchecked.h>. That is,
#include <stdchecked.h>
void read_next(int *b, int idx, checked int *out) {
int tmp = *(b+idx);
*out = tmp;
}Ok, we've squeezed in an 'r'.
> integer under/overflow
very few languages actually solve this completely. you'd need refinement types to do so.
The problem here is you’re not writing C, you’re writing a language that kind of resembles C but is nonportable. So you might as well just use a different, memory safe language at that point.
I think you're missing an important point:
Checked C’s design is distinguished by its focus on backward-compatibility, incremental conversion, developer control, and enabling highly performant code
The problem is that there is a huge body of decades-old C code out there that we all depend on.
A typical Docker container with a web app contains maybe 10K lines of your own code in Python/Ruby/JS, 100K lines of framework code, and 1M or 10M lines of C code (i.e. the interpreter itself, the web server, SSL, the base image, etc.)
So I welcome new languages focused on incremental conversion.
That's what I'm doing with Unix shell: http://www.oilshell.org/blog/2018/01/28.html
I applaud projects like Corrode, a C to unsafe-Rust translator, although undoubtedly they would have an easier time if Rust was designed in the first place for conversion.
https://github.com/jameysharp/corrode
So Checked C seems like a great idea to me.
I've noticed that most programmers seem to wildly overestimate the rate at which code gets rewritten. I think it's more accurate to say that code piles up over time. And unfortunately when the foundations are unstable, what you build on top is also unstable.
There is also a new tool, c2rust, https://c2rust.com that I believe was inspired by Corrode.
In addition, we also have the bindgen tool in Rust that will auto generate the Rust FFI from C header files. It works really well. At that point you wrap the FFI in more Rust idiomatic interfaces, and expose those as the preferred library for others to use.
I mention this, because it is a path toward allowing for the carrying forward of a C codebase, but using a safe language like Rust for all future work, perhaps going back and rewriting the C if it’s deemed important.
There are many examples of this pattern being applied to projects.
In addition to that, there is a cbindgen tool, that does the inverse of the bindgen tool, which would allow you to write Rust code, and quickly generate C header files for the FFI, allowing for critical areas of C to be replaced with Rust, without converting the entire project.
Putting all of that together might be a reasonable thing for large projects to convert iteratively, should they decide it’s an important thing for them to do.
That is nice for userspace apps, but no one is going to use c2rust on Aix, HP-UX, *BSD, Linux, Zephyr, QNX, RTos, ....
I think something like Checked C is the only way to actually make it work on those use cases.
No one is going to use Rust on any of these projects, because of political reasons and portability concerns (because LLVM doesn't target all platforms). But those two issues also apply to Checked C and I doubt any project you listed will ever use it either.
Since BSD's system compiler is now LLVM, I think it's only a matter of time some enlightened BSD variant (OpenBSD maybe?) tries something like Checked C.
I don't know what programming language or extensions they will ever use, but this year's Linux Security Summit was pretty clear that are lots of work to do regarding Kernel Self-Protection Project still need to be done.
And that is just one UNIX variant.
Ziglang is also an incremental update to C. No FFI needed, can directly import C header files.
It's not safe though. ATS/Rust are.
Yes. This is why people put so much effort into projects like Valgrind [1] and american fuzzy lop [2]. Preserving portability by keeping the same language and trying to find as many bugs as possible using clever tooling.
It's about the control you can have. You can gradually add in annotations without having to rewrite the whole program as well as being able to selectively check buffer-overflows (i.e. checking everything vs checking code regions that you don't have security guarantee but would like to have). Additionally they also have features that allows you check overflow statically on limited cases.
No, the problem here is that no one is going to re-write UNIX derived OSes into something else, nor change their culture around C.
So there are only two options left, make C safe in some way no matter what, or just throw everything away.
The Linux Security Summit 2018 set of talks regarding kernel mitigation exploits is pretty clear how bad security exploits due to C are turning out to be.
The thing is, C can not be made "safe" without breaking old code and specified semantics. Even if you for example follow all the strict aliasing rules, you can still 1) make a mistake 2) access/modify arbitrary memory. Then you have overflows, threading races ... It is ok, C is meant to be like that. You can make C safer, make things better (arguably), that is a different thing. And it is beeing done, slowly and without breaking (in theory) stuff.
I agree the solution is not perfect and ideally I would nuke the language out of orbit.
But that approach will never win the hearts of UNIXy kernel devs, and I guess until Quantum computers take over we are past the point of having alternative OSes to UNIX based ones, because no one is going to spend the required money to do that.
So a migration path towards safety is required.
I think, eventually, the whole stack will get rewritten from the ground up. Someone is going to come up with a minimal OS that can be run on virtual machines that implements LLVM in a provably correct way, the OS hooks will get reimplemented as libraries for all the languages, and over a few years or so the entire cloud infrastructure will simply switch over.
I think we're already half the way there, Amazon Linux is how widely deployed already? The surface area between application logic and OS code is shrinking by the year, once serverless goes mainstream then even application code will start to get rare, pushed to only those specific projects that require human interaction. Everything else is 'mere' computation, something that can be specified purely algorithmically and executed by anything.
The OS will only be relevant in the context of personal computing, the human interface to the machine kingdom which shed those trappings decades ago.
20-30 years down the road I see a translation layer unifying all three major, now solely personal, operating systems, and a fundamental, free software core being driven out, eventually supplanting proprietary development. Apple will do its best to retain tight coupling between their hardware and software, but they'll slowly fade just like Microsoft did as programming itself is revealed to consist of two main activities, intricate UX and comparatively dumb data plumbing, with a small priesthood of hardware driver maintainers / database developers for those who still want to do 'real' programming.
Everybody will be able to code at some level and it'll be amazing.
That's very interesting. Could you share more of how you envision this system working?
At some point, backwards compatibility loses its grip on necessity. At this point, reinvention becomes possible. You can look at various parts of the tech ecosystem and sort it all into categories based on just how much the human users are depending on old behavior.
Something like vim has archaisms that are only mildly depended on, allowing a project like neovim to credibly call itself the future. But Python 2 still has a large install base of software that's still running it. Python 3 didn't offer a smooth enough upgrade path from Python 2, so we're seeing the slow bifurcation of the community into two language. This also happened with Rails 2, there is still large enough install base of Rails 2 software that there's a dedicated effort to support it, Rails LTS.
These bifurcations will last until gradual enough upgrade paths are made and failure modes are isolated enough so that moving a project off of old tech and onto new tech can be accomplished without business investment. It's merely another user story to be put in the pipeline. Old applications will slowly get either abandoned or migrated or reinvented over time.
This process fails when you consider underlying infrastructure, which by its nature largely has a much deeper base of software that depends on it. Applications can have many dependencies, but those dependencies tend to be well-enumerated, as not enumerating them leads to a maintainability nightmare for people unfamiliar with the system. And nothing but other business software depends on the application.
But infrastructure code's dependency structure is inverted. Many more applications depend on infrastructure and infrastructure tends to have comparatively fewer dependencies of its own. When infrastructure has lots of dependencies it makes it harder to scale, and scale is the whole point of infrastructure. This need for infrastructure to all 'hang together' so as to provide for scale is what's behind the modern concept of an operating system. Applications tend to rely first on the software stack, second on the operating system. It's much easier take a Rails application built to run on Ubuntu and migrate it so it can run on Red Hat or Docker than it is to get it to run on Sinatra, god forbid another general purpose programming language. In fact, it's so much easier that it's considered a major red flag for this to not be the case. Operating systems are more fungible than software stacks.
This process of bifurcation followed by slow abandonment of harder-to-maintain systems in favor of easier-to-maintain ones is just going to slowly reach its apotheosis when line of business software that is much of a career programmer's bread and butter is going to slowly lose all attachment to, not just the operating system, but also the software stack as well. FaaS, Function as a service, is something you can add to your application right now, and once you do that work once, it just gets easier and easier to offload portions of the application that simply glue things together or compute things, to a magical land where teams of elves do literally everything else for you for a small fee, is going to become mainstream sooner rather than later.
> I agree the solution is not perfect and ideally I would nuke the language out of orbit.
> But that approach will never win the hearts of UNIXy kernel devs
I don't think this is fair. The Linux kernel contains (as of writing) 17913052 lines of C. If you have a plan to rewrite 18 million lines of code and patch every single dependency, fine. But until then such concerns will dominate.
It's also worth noting that the Rust many people write is not the low-level stuff that C programmers and systems programmers write. See https://www.cl.cam.ac.uk/~srk31/research/papers/kell17some-p... for a better explanation of that.
I don't have such plan, but Linux kernel devs have some mitigation plans.
Not to rewrite C into something else, but to get out of the current escalation of kernel exploits, now that Linux is a commodity OS, it gets targeted as such
https://events.linuxfoundation.org/events/linux-security-sum...
Hence Kernel Self Preservation Project, removing VLAs from kernel code, adding all possible ways of mitigation, and hopping that eventually tagged pointers like SPARC and ARM have become common across all supported hardware.
The charts shown by Google employees regarding exploits are quite telling.
Do you see the difference between "Here's a very small change, easy to implement and add." and "Let's just rewrite the whole thing and buy new devs" ?
Yes, I do. But the unfortunate reality is that your "small change" makes your code much harder to work with for other people, since you are now writing a C derivative instead of standard C.
This is the same argument that was used when Microsoft launched Typescript. As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption. So it seems to be a pretty solid strategy, and if it brings people closer to a safe language, even if not 100% safe, it's still an improvement on the status quo.
> As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption
Did it? Adoption seems to me to be a pretty silly metric given all the politics of language choice.
Given that the goal is getting C developers to adopt best practices, adoption is the most relevant metric here.
Well, probably most C code is not strictly standard-compliant C, because of undefined behaviour, GNU extensions, or whatever else.
Completely agree. It's interesting to see Microsoft extend C and others, here, are embracing it so quickly.
That's an interesting choice of words you have there. Are you trying to allude to what I think you're alluding to?
Checked C source code:
https://github.com/Microsoft/checkedc
I wonder how this will work out. GCC had checked fat pointers for years, used by almost nobody. Microsoft can make it stick on their platform, because they own the libraries and can make them use fat pointers.
I hope this works out better than Microsoft Managed C++.
Managed C++ has been replaced by C++/CLI in 2005 and it pretty much alive.
Its purpose was to make .NET interop easier with C++ libraries and has been quite successful at it, instead of having to play around with P/Invoke until everything works properly.
Regarding Windows, Microsoft is pretty much on the ongoing transition to C++, where these kind of issues are easier to tackle.
I bet their goals regarding Checked C have more to do with the adoption of UNIX (Linux/BSD) based technologies for Azure deployments and IoT devices like Azure Sphere.
There has been dozens of "safe" additions, variants and extensions to C along the years (from commercial companies, from individuals, from university labs, ...). Not one of them ever took off. This one did not either.
We always just say "hmmm... right... this one may look a bit better than the previous one (or doesn't)" and we never bother to use them. After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
> After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations. Inevitably, tests never catch all issues and you end up with type errors and segfaults in production.
> This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations.
In the case of a "safer C" I suspect it's largely because it never materialized. C still has its deficiencies relative to Rust, but Rust is the first viable competitor.
Sure it did, all major systems languages outside UNIX universe were safer than C.
But the OSes they were tied to were killed by UNIX's adoption, just like the browser has helped JavaScript to take over areas of other languages.
For starters people have to realize that safe C is not a research problem. It's an engineering problem of compilers, package management and C software. And all it takes to take off is some engineering work.
However funding that work is a big problem, since it mostly benefits others.
You missed the most important one, people.
Mentalities are very hard to change and we all know that only lousy programmers make errors in C.
As far as I know, Managed C++ is mostly used as an interop layer between C/C++ and C#, sort of like the inverse of P/Invoke.
Since P/Invoke can't marshal C++ classes, Managed C++ gives you a way to expose such things. It's a very special-purpose chimera.
October 1, 2018 ? :O
That's likely when the paper will be presented at the https://cybersecurity.ieee.org/blog/2018/01/31/ieee-secdev-2...
A buffer overflow?
There is some precedent for a new language that gradually improves safety of an existing language; TypeScript did this to JavaScript and is arguably successful.
I don’t know if Checked C will have the same success story but if devs can easily add this to their existing projects and it catches bugs, that seems like a win to me.