▲ 220 points
back
89 comments
"eBPF Documentary: An exciting train wreck in progress"
That would be a better title. eBPF started as a small extension to just be able to insert small trivial hooks. It's now basically a hacked-up broken WebAssembly clone, with zero forethought put into it. NIH syndrome at its worst.
It has recently grown unlimited loops with runtime metering, making the static verifier basically a worthless complexity. Before that, it had acquired exceptions and stack unwinding.
While I agree its design isn't the best, versus something like Solaris DTrace, it predates WebAssembly, so hardly a clone.
eBPF itself is indeed ancient. However, the explosion of NIH-ed functionality is not. Most of it happened after the standardization of WASM.
Bytecode based execution goes back to the 1960's, yet another thing that WebAssembly advocates tend to forget when writing comments and blog posts about its greatness.
Architecture independent eBPF "rescue binaries" could be an interesting part of a distribution toolset.. or attack rootkit. It's hard to have nice things.
It seems that reasonable changes - well vetted ones - seldom happen. What to do?
It is a pretty common pattern now that if you allow someone to take on a "task" without guardrails it just explodes into nonsensical feature creep.
Also in article we can read "like putting JavaScript into the Linux kernel" :rotfl:
The genius of ebpf is allowing for pluggable policy in a world where the kernel API is very slow to change and can’t meet everyone’s needs. Whether it’s how the kernel handles packets off the wire, how it controls traffic, scheduling entities, or instrumentation, ebpf lets you provide logic rather than turn a bunch of knobs or use a bespoke syscall that only handles one case. It also moves the processing logic to the data in the kernel rather than having the kernel have to do expensive copies to and from userspace.
ebpf isn’t really novel beyond the interfaces it provides. They are just kernel modules that have been vetted and are sandboxed. Inserting executable code has been part of the kernel since forever in module form and kprobes.
> ebpf isn’t really novel beyond the interfaces it provides. They are just kernel modules that have been vetted and are sandboxed. Inserting executable code has been part of the kernel since forever in module form and kprobes.
This should be sung from the mountaintops. This concisely summarizes nearly everything that uninformed reader should take away from the comment section.
> the kernel API is very slow to change and can’t meet everyone’s needs
Better yet - eBPF provides a stable ABI:) It makes things that were formerly kernel-internal possible to work with from a stable ~userspace interface.
bpf tooling generally provides no stability guarantees when you interact with kernel primitives. See [0], for example. Tho things have improved somewhat with CO-RE
[0] - https://lore.kernel.org/lkml/93a20759600c05b6d9e4359a1517c88...
I'm curious what this guarantee includes - the bytecode? Because the actual in-kernel eBPF API is famously unstable, with eBPF-based applications usually requiring a cutting-edge kernel version (for industry anyway). And of course the eBPF programs themselves rely on accessing structures for which no stability guarantees are made whatsoever.
I've been hearing more and more about eBPF, especially here on HN.
I haven't yet watched the documentary so perhaps it is answered there. But, the analogy of JavaScript inside the kernel is great and I'm left wondering: what was the way to do it previously? Userland network tool? This standardizes on a interface to the kernel, not a language, right? Feels off to say it is JavaScript because that comes with a lot of baggage, but also (as a versatile and ubiquitous language) incredibly powerful and useful tool. Is that intentional by the author?
> But, the analogy of JavaScript inside the kernel is great and I'm left wondering: what was the way to do it previously? Userland network tool? This standardizes on a interface to the kernel, not a language, right?
Guess/sketch: It's a language in most senses. Previously the kernel had APIs for packet filtering rules for iptables etc., but the set of rules you could use was somewhat "static" - rules would have parameters, so you could do things like if the source IP is in this range then rewrite it as this and direct it to this interface, but it was kind of like one of those visual flowchart languages where you can drag and drop the available boxes in a given order, but if there isn't a box to do what you want then you're stuck. Whereas with eBPF it really is scriptable - rather than a specific rule type you can just submit the script you want it to run - and nowadays it's become kind of a general kernel scripting language rather than just for networking.
I'd draw a parallel with how 3D graphics programming has shifted from "you can do these kinds of transformations, submit a list of what you want to run in what order" to "this is our shader programming language, just write whatever you want to do as a program in this language".
iptables is definitely limited in comparison to eBPF, but that isn't the innovative step. BPF was around for more than twenty years before eBPF came around. Around 2013 I worked on a packet analysis pipeline that generated BPF code dynamically at runtime. eBPF isn't more scriptable than BPF in this sense. The language does add some opcodes and loops that weren't available in the original, but this is relatively modest.
The real genius of what these folks did was extending the usefulness of BPF beyond the network stack. Without a provably safe language it would've been impossible to enable flexible kernel tracing.
> This standardizes on a interface to the kernel, not a language, right?
It's a VM that runs JIT-compiled eBPF programs. You can write code in C or Golang or other languages that compiles down to eBPF. I did a video looking at the kernel eBPF code here: https://youtu.be/hznUH_zP77U?t=1165
> wondering: what was the way to do it previously? Userland network tool?
My understanding is that userland network tools were common in fields like finance that needed fast custom networking and wanted to eliminate the overhead of context switching. I don't know how common they are/were in other fields though.
golang compiles to eBPF? including the whole golang runtime with garbage collector? that... surprises me
edit: I don't see anything on the web that would show golang compiles to eBPF. I see bpf2go which is the other way around.
Ad-hoc kernel extensions were a pretty common answer, and one thing a lot of people love about eBPF is that it subsumes most of the reasons people wrote lkms commercially.
One of the big wins is not so much “build and run your own stuff” but there are very nice low-cost (in terms of compute) performance utilities built on eBPF
https://github.com/iovisor/bcc
There are so many utilities in that list; there’s a diagram midway down the readme which tries to help show their uses. bcc-tools should be available in any distro.
Also, Brendan Gregg does a ton of performance stuff that is worth knowing about if you check out his other work. Not eBPF only. Flame graphs are useful.
> what was the way to do it previously?
SystemTap, and it still is. BPF can only do so much.
I was a little disappointed DTrace[0] was not mentioned at all. The instrumentation (not the SDN) isn’t novel, not even to Linux (DTrace is available on Linux - I understand licensing is at least questionable (for some distros), but that aside…). [1]
[0] https://en.wikipedia.org/wiki/DTrace
[1] https://docs.oracle.com/en/operating-systems/oracle-linux/dt...
DTrace was not the inspiration for eBPF at all, so it's not obvious it is relevant to mention. As the documentary mentions, the initial impetus for eBPF, was software defined networking.
eBPF is a much bigger and more comprehensive infrastructure piece (networking, tracing, security, etc) than DTrace. And thanks to licensing issues, even within the limited domain of tracing, DTrace will likely become a footnote in history, while eBPF becomes available on every major OS platform.
There aren't "licensing issues" with DTrace -- you are merely referring to the fact that it is licensed under the MPL-derived CDDL and not the GPL. But it is definitely true that they are not seeking to solve the same problems! Safety is very core to DTrace[0]; the difference here is entirely deliberate.
i feel like the fact that the page mentions brendan gregg is a pretty strong reference to dtrace already. actually it doesn't just mention him, it's completely written by him
I think people that know, know he was involved (in the userland aspect) of DTrace, but I get a sense for some reason there’s no love lost between Brendan and “DTrace”.
AFAICT DTrace is more a parallel branch. Both inspired by cBPF.
What should get more talk IMO, is the exokernel XOK's kernel VMs which went way harder than even eBPF does towards user space programmability for the kernel as a core primitive.
For instance instead of sleep(2) or futex(2) calls, XOK exposed "wake programs" that user space would register for the scheduler to run to answer "is this blocked thread runnable again". Would have solved the collabra's need to change futex(2) to work more nicely with wine/windows primitives in a more general way.
eBPF modules can be closed-source, right? I can see a future where things like ZFS are paid eBPF modules, or games ship with proprietary anti-cheat eBPF modules.
In theory, yes. In practice pretty much all useful applications need to call kernel APIs that mandate the eBPF code being GPL.
You can't call kernel APIs from eBPF without triggering the license, I suppose?
I saw the airing of it at Kubecon -- also met the Finnish guy there in the video. I mistook him for another Finnish guy with blond hair. Didn't meet the Russian though.
It makes me wonder, what was the point of introducing Rust into the kernel when they already have eBPF?
Couldn't they have just used the existing eBPF compiler for creating safe kernel code?
Maybe the distant future of the Linux kernel is for almost everything to compile down to eBPF bytecode except the part that runs the bytecode.
That's a cool conjecture but it kind of misses the point of eBPF.
The point of it is that you can run user-defined programs while avoiding the costly context switch between user space and kernel space.
The kernel already is the kernel. Compiling kernel code to eBPF programs would offer seemingly no performance gains, since you're already in kernel space; there is no costly context switch to avoid.
ebpf seems to be a very interesting idea and have been experimenting with it. Still I find it weird that we're doing documentaries on software "frameworks"
"Documentary" may be a bit of a strong word. It's more of an image film / ad for Cilium and/or the ebpf foundation.
Brendan Gregg will get a Turing Award for this, congrats!
A Turing Award for avoiding a context switch?
I mean, I love eBPF more than most, but this is a practical engineering solution to a logistical problem that didn't really need to exist in the first place.
This is not genius and not an order-of-magnitude improvement to an important computer science problem; it's an improvement to a costly artifact of the Linux kernel.
It has always seemed quite obvious to me that dealing in machine code is a flawed approach for distributing software. At the most basic level, it entails giving someone else near unfettered access to the hardware of your computer and simply hoping that they do nothing malicious or malformed. Yet the software world as a whole seems continually shocked at the idea of using anything else. Perhaps someday we will learn this lesson in its entirety and begin to share code rather than blobs.
I don't think machine code is the problem? Running a native binary doesn't give it access to anything inherently; the OS gives access through syscalls, and can impose restrictions - and indeed, does; it's not like running a binary on Linux automatically gives it access to anything under /dev.
Also, it's not that the whole software world thinks blob-only software is normal; I'm typing this on a nice comfy GNU/Linux box where the only blobs are some firmware. (Edit: And to be quite clear, a good chunk of this community would really like to get rid of those blobs too, it's just that we don't have a fix at this point.)
> Running a native binary doesn't give it access to anything inherently
While this is true from a certain perspective, machine code creates a system which must grand access to many things to become usable. A shared file system is a good example of this. Some software could easily echo a line into you .profile that tries to launch a key-logger, and this works in many cases. The expectation of software existing as opaque files creates a huge amount of work for the OS in verifying the exact behaviour of the software as it runs (and in ways which can often be circumvented), rather than a source-based approach in which malware is never allowed to touch the processor.
> I'm typing this on a nice comfy GNU/Linux box where the only blobs are some firmware
So you suffer the worst of both worlds then. You've had to download and compile the source yourself, but as the software is designed around being distributed as blobs, so you enjoy none of the benefits that might come from source distribution.
Android and mobile platform make it hard to distribute as fully compiled source.
"That's a really cool security aware script language you've got there! So.. um.. how can I extend it to call third party libraries?"
Perhaps the idea that "the computer" is one single entity with a shared security domain and view of hardware is the flaw. Why can my web browser read my tax documents unless I go through a bunch of rather absurd efforts to prevent something so simple?
Because you want to be able do report your taxes documents to the tax office? It's one of those things that sound so simple on paper, but every time someone does that trivial thing and not make documents available to the web browser, usability suffers.
The real answer why the browser can read certain files is much more complex, your web browser is not a singular entity anymore. And the network and protcol speaking parts of it can't access your documents, according to the principle of least authority.
It's far from perfect and gets hacked every time, but do take the time to read how that's done. The hacks are just as complex as the web browser itself. The practical problem with the browser is the enormous complexity of functions, everything from OpenGL to databases to p2p and usb, that keeps growing boundlessly.
ChatGPT?
eBPF allows user-defined programs to run in the kernel.
This is huge for performance-sensitive code that executes against network packets: you don't have to context switch between kernel space and user space.
It's worth pointing out Solana's extreme competitive advantage over other chains is almost entirely due to it running on a variant of eBPF. †
This is an order-of-magnitude leap over other implementations and essentially the way you should do it, if you were to write it from scratch, aside from special purpose hardware fabrication.
† The second reason Solana is so fast is extreme parallelism: all accounts that are used in a transaction must be marked as either "read-only" or "writeable" before sending the transaction, allowing the runtime to parallelize all reads and only solve write contention when necessary.
This comment seems like half GPT nonsense and half unsubtle advertisement for whatever Solana is.