Item — HN Reader

Competition is good (not everyone using Linux I mean), and I've ran FreeBSD on my desktop and server for a few years.

But whenever Netflix's use of FreeBSD comes up I never come away with a concrete understanding for: is the cost/performance Netflix gets with FreeBSD really not doable on Linux?

I'm trying to understand if it's inertia, or if not, why more cloud companies with similar traffic (all the CDN companies for example) aren't also using FreeBSD somewhere in their stack.

If it were just the case that they like FreeBSD that would be fine and I wouldn't ask more. But they mention in the slides FreeBSD is a performance-focused OS which seems to beg the question.

6 comments

The community is such that if one of either FreeBSD or Linux really outperformed the other in some capacity, they'd rally and get the gap closed pretty quickly. There are benchmarks that flatter both but they tend to converge and be pretty close.

Netflix has a team of, I think it was 10 from the PDF, engineers working on this platform. A substantial number of them are FreeBSD contributors with relationships to it. It's a very special team. That's the difference maker here. If it was a team of former Red Hat guys, I'm sure they'd be on Linux. If it was a team of old Solaris guys, I wouldn't be surprised if they were on one of the Solaris offsprings. Then, Netflix knew that at their scale and to make it work, they had to build this stuff out. That was something they figured out pretty quickly, they found the right team and then built around them. It's far more sophisticated than "we replaced RedHat Enterprise with FreeBSD and reaped immediate performance advantages." At their scale, I don't think there is an off the shelf solution.

I think you're close but missing an element. FreeBSD is a centrally designed and implemented OS from kernel to libc to userland. The entire OS is in the source tree, with updated and correct documentation, with coherent design.

Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix. The reasons for design choices going all the way back to freebsd 1 are available on the mailing list in a central place. Everything is there.

Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

The resources to match the relatively small freebsd design and implementation team are minuscule compared to the infinite crawl of linuxes, a team of 10 freebsd sysprogs is basically the same amount of people responsible for designing and building the entire thing.

It comes down to the sources of truth. In the world of fbsd that's the single fbsd repo and the single mailing list. For a linux, that could be thousands of sources of truth across hundreds of platforms.

> infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

Linux 2.2 is from 1999. It can barely do SMP. That's pretty much a crazy person claim.

Googlers say ProdKernel is merged forward every few years:

> Every two years or so, those patches are rebased onto a newer kernel version, which provides a number of challenges.

https://lwn.net/Articles/871195/

> Every ~2 years we rebase all these patches over a ~2 year codebase delta

https://events.linuxfoundation.org/wp-content/uploads/2022/0... (and many other places)

I understand the point you're trying to make and I agree that FreeBSD tends to have cleaner code and better documentation at different levels but I don't think that it makes it that much more difficult. If you dropped in from a different world and had zero experience then I think you're right and a systems team would almost always pick BSD. Any actual experience pretty quickly swings it the other way though; there are also companies dedicated to helping you fill in those gaps on the Linux side.

I've built a couple embedded projects on Linux, when you're deep on a hard problem, the mailing lists and various people are nice, but the "source of truth" is your logic analyzer and you debug the damn thing. Or your hardware vendor might have some clues as they know more of the bugs in their hardware.

Fair points taken, i was a bit zealous in my use of the word any, the word many is more correctly applicable.

In regard to sources of truth, i mean from the design considerations point of view. For instance, why does the scheduler behave a certain way? We can analyze the logic and determine the truth of the codes behavior but determining the reason for the selection of that design for implementation is far more difficult.

These days yes, off the shelf linux will do just fine at massively scaling an application. When netflix started building blockbuster was still a huge scary thing to be feared and respected. Linux was still a hobby project if you didn't fork out 70% of a commercial unix contract over to rhel.

The team came in with the expectation and understanding they would be making massive changes for their application that may never be merged upstream. The chances of getting a merge in are higher if the upstream is a smaller centralized team. You can also just ask the person who was in charge of the, let's say for example, the init system design and implementation. Or oh, that scheduler! Why does it deprioritize x and y over z, is there more to why this was chosen than what is on the mailing list?

The pros go on and on and the cons are difficult to imagine, unless you make a vacuum and compare 2024 linux to 2004 linux.

> Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix

If any team would prefer this, then as gp asked: why is Linux overrepresented at other Non-Netflix content-delivery shops?

> infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

Say what?

They are trying to get off or have gotten off their kernel fork called "Prodkernel" for some time now.

https://lwn.net/Articles/871195/ https://events.linuxfoundation.org/wp-content/uploads/2022/0...

And ProdKernel generally lags mainstream by only a few years, as can be seen in the things you linked to.

The change being talked here is moving from merge ~2 years to merge all the time. Saying they're stuck on something from 1999 is ridiculous.

Practically this just doesn't matter all that much. You can prefer one approach to the other and that's all fine, but from a "serious engineering" perspective it just doesn't really matter.

> Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

2.2? Colour me skeptical on that.

But it's really not that hard to make a Linux distro. I know, because I did it, and tons of other people have. It's little more than "get kernel + fuck about with Linux booting madness + bunch of userland stuff". The same applies to FreeBSD by the way, and according to the PDF "Netflix has an internal “distro”".

The problems Google has is because they maintain extensive patchsets, not because they built their own distro. They would have the same problems with FreeBSD, or any other complex piece of software.

It's hard to do apples to apples comparison, because you'd need two excellent, committed teams working on this.

I'm a FreeBSD supporter, but I'm sure you could get things to work in Linux too. I haven't seen any posts like 'yeah, we got 800 gbps out of our Linux CDN boxes too', but I don't see a lot of posts about CDN boxes at all.

As someone else downthread wrote, using FreeBSD gives a lot of control, and IMHO provides a much more stable base to work with.

Where I've worked, we didn't follow -CURRENT, and tended to avoid .0 releases, but it was rare to see breakage across upgrades and it was typically easy to track down what changed because there usually wasn't a lot of changes in the suspect system. That's not really been my experience with Linux.

A small community probably helps get their changes upstreamed regularly too.

The truth is that some senior engineer at Netflix chose to use FreeBSD and they stick to that idea since then, FreeBSD is not better it's happen to be the solution they chose.

All the benefits they added to FreeBSD could be added the same way in Linux if it was missing.

YouTube / Google CDN is much bigger than Netflix and runs 100% on Linux, you can make pretty much everything work on modern solution / languages / framework.

Sorry, this is seriously misinformed:

> YouTube / Google CDN is much bigger than Netflix

Youtube and Netflix are on par. According to Sandvine, Netflix sneaked past Youtube in volume in 2023[1]. I believe their 2024 report shows them still neck-and-neck.

> you can make pretty much everything work on modern solution

Presenting a false equivalence without evidence is not convincing. "You could write it in SNOBOL and deploy it on TempleOS". Netflix didn't choose something arbitrary or by mistake. They chose one of the world's highest performing and most robustly tested infrastructure operating systems. It's the reason a FreeBSD derivative lies at the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn Darwin and thence macOS, iOS etc.

It continues to surprise me how folks in the tech sector routinely fail to notice how much BSD is deployed in the infrastructure around them.

> All the benefits they added to FreeBSD could be added the same way in Linux

They cannot. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel. Netflix is touting here the value of FreeBSD's unified release, since the bisect process fundamentally relies on a single dimension of change (please ignore the mathematicians muttering about Schur's transform).

[1] https://www.sandvine.com/press-releases/sandvines-2023-globa...

> There's a reason it's the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn macOS.

Licensing?

FreeBSD is a fine OS that surely has some advantages here and there, but I'm also inclined to think that big companies can make stuff work if they want to.

PHP at Meta seems like a pretty good example of this.

The permissive, freewheeling nature of the BSD license is touted by some as an advantage for infrastructure use but in practice, Linux-based devices and services aren't particularly encumbered by the GPL, so to me it's a wash.

Could be lawyers at some companies just didn't want anything to do with the GPL, especially if they're fiddling with the kernel itself. Maybe they're not even correct about the details of it, just fearful. "Lawyers overly cautious" is not an uncommon thing to see.

Having seen the screams from some people when you use GPL legally I can't blame the lawyers. You might be correct but you will still get annoying people yelling you are not. there is also the cost to verifying you are correct (we put the wrong address in for where to send you request for sorce, fortunately a tester caught it before release, but still expensive as the lawyers forced the full recall process meaning we had to send a tech to upgrade the testers instead of saying where to download the fix)

> […] Linux-based devices and services aren't particularly encumbered by the GPL, so to me it's a wash.

Linux uses GPL 2.x. If we are talking about GPL 3.x things may be different:

> One major danger that GPLv3 will block is tivoization. Tivoization means certain “appliances” (which have computers inside) contain GPL-covered software that you can't effectively change, because the appliance shuts down if it detects modified software.

* https://www.gnu.org/licenses/rms-why-gplv3.en.html

If you want to ship (embedded) GPL 2 Linux-based devices things are fine, but if there's software that is GPL 3 then that can cause issues. Linus on GPLv3:

* https://www.youtube.com/watch?v=PaKIZ7gJlRU

> Licensing?

I think you've hit the nail on the head. I used to work for a very very large company. And they had a very strong preference for BSD licensed software, if we were building something using outside software. A few years ago, Stever Balmer and others spent a lot of time calling spreading FUD by calling Linux and GPL a "cancer". Believe it or not, that stuff had a massive impact. Not on small shops, but large company lawyers.

Over the years, Steve left Microsoft and Microsoft has become a lot more Linux friendly. And the cancer fear has subsided. But it was very very real.

On a side note, if I recall correctly, Steve Jobs wanted to use Linux for MacOS. But he had licensing concerns and called up Linus and asked him to change a few things. Linus gave him the middle finger and that is how we got MacOS with BSD.

I doubt this story about Linus and Steve. MacOS is NEXTSTEP-derived and that pre-dated Linux by several years.

Fun fact! NeXT was a commercial att unix fork. The transition from the unix base to the BSD base did infact happen at apple after the acquisition. The value of next was in its application library, which would eventually become the mac foundation libraries like coreaudio and cocoa etc. The earliest releases of Rhapsody are very illuminative about the architecture of XNU/OSX. I don't doubt that linux was considered. There's a specific time when the actual move of rhapsody to a freebsd base occurred and it was at apple sometime in 97 or 98 iirc.

NeXTSTEP was always BSD4.3-Tahoe Unix, not AT&T Unix.

> On a side note, if I recall correctly, Steve Jobs wanted to use Linux for MacOS.

Steve offered Linus a job, using Linux was never on the table.

https://www.theverge.com/2012/3/22/2893581/linus-torvalds-li...

It's a mix of licensing and performance/stability. The licensing ensures that a company can implement their own IP and not have to contribute that back to the community as open source.

The performance/stability of FreeBSD is no joke, that's why, as mentioned previously, storage vendors like Isilon and Netapp choose FreeBSD as the base. They can contribute upstream when needed, but they don't have to provide any source for how their storage appliance software.

https://freebsdfoundation.org/testimonial/isilon/ https://freebsdfoundation.org/end-user-stories/

Be all that as it may, this is still a “why not Linux” line of thinking, rather than “why FreeBSD”, which is the more interesting question. And it is not a binary choice.

Because FreeBSD is a perfectly fine OS for server type stuff, and maybe it's better for this, that, or the other thing. So someone probably picked it and it has worked ok for them.

Just to be clear, you don't need to convince me - I've been preferring BSDs for my own infrastructure since the 90s. But I'm mindful that the discussion can quickly become polarised and winds up with negative framing or binary assumptions, which isn't helpful for others asking the same question.

FreeBSD or Linux management plane on routers/switches is mostly irrelevant. The majority of the data simply flows through the asics untouched.

> Incorrect. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel.

You can certainly bisect the Linux kernel changes though. And the bug in question was a kernel bug. For a project like this, IMHO, most of the interesting bugs are going to be kernel bugs.

Perhaps so, but this knowledge is post hoc for the incident and does not undermine the engineering value of the unified release.

Probably, if you were doing this on Linux, you'd follow someone's Linux tree, and not run Debian unstable with updates to everything.

You might end up building a lightweight distribution yourself; some simple init, enough bits and bobs for in place debugging, and the application and its monitoring tools.

Anyway, if you did come across a problem where userland and kernel changed and it wasn't clear where the breakage is, the process is simple. Test new kernel with old userland and old kernel with new userland, and see which part broke. Then bisect if you still can't tell by inspection.

> It's the reason a FreeBSD derivative lies at the core of Juniper routers

Juniper is a particularly poor example to start the list with as they have been moving away from FreeBSD for years now with Junos OS Evolved being Linux based and specifically about how being based on FreeBSD was no longer a benefit to them.

no one seriously uses Evo in the way JunOS powers things. they’ve managed to package FreeBSD inside some Linux flavors. it’s wild, imo.

Guess I'm nobody, I think the PTX line is great ¯\_(ツ)_/¯.

>It's the reason a FreeBSD derivative

A reason. Licensing also plays a role. Some may say the most important one.

>Case in point

Not really. This is an advantage, yes, but was inherent to BSD development style. Not an addition they did. Assume GP refers to other presentations, talking about the optimizations needed to get the performance Netflix needs from FreeBSD.

>doesn't exist for Linux

But can be done by putting kernel and userland in same repo as modules.

'runs 100% on Linux' is a bit vague. What customisations do they do?

Google has ~9000 kernel patches, and a fairly custom userspace. Saving 2% at that scale is huge.

My understanding was that Netflix liked FreeBSD for a few reasons, some of them more historical than others.

* The networking stack was faster at the time

* dtrace

* async sendfile(2) https://lists.freebsd.org/pipermail/svn-src-head/2016-Januar...

Could they have contributed async sendfile(2) to linux as well? Probably. In 2024 these advantages seem to be moot: ebpf, io_uring, more maturity in the linux network stack plus FreeBSD losing more and more vendor support by the day .

Might be a manpower thing. By hiring a bunch of FreeBSD Core devs, Netflix might be able to get a really talented team for cheaper than they might get a less ideologically flavored team. (I say this as I set up i3 on FreeBSD on my Thinkpad X280, I'm a big fan!)

They also get much more control. If they employ most of the core FreeBSD devs they basically have their own OS that they can do what they like with. If they want some change that benefits their use case at the detriment of other people they can pretty much just do it.

That's not really possible with Linux.

The flip side is of course that in this scenario they would have to finance most improvements. In Linux land the cost of development is shared across many, which one might expect to yield a generally better product.

When netflix was founded the only viable commercial linux vendor was rhel and the support contract would have been about the same as just hiring the fbsd core team at salary.

People really do not remember the state of early linux. Raise a hand if you have ever had to compile a kernel module from a flash drive to make a laptops wifi work, then imagine how bad it was 20 years before you tried and possibly failed at getting networking on linux to work, let alone behave in a consistent manner.

The development costs were not yet shared back then, most of the linux users at the vendor support level were still smaller unproven businesses and most importantly if you intend to build something that nobody else has you do not exactly want to be super tied to a large community that can just take your work and directly go clone your product.

Hiring foss devs and putting them under NDA for everything they write that doesn't get upstreamed is an excellent way to get nearly everything upstreamed aswell, and the cost of competitors then porting these merged upstream changes back down into their linux is not nothing, so this gives a competitive moat advantage.

Netflix, the online service, launched in 2007. The previous business, doing DVD mailing, had no such high bandwidth serving requirements. You're exaggerating the timeline quite a lot.

> When netflix was founded the only viable commercial linux vendor was rhel and the support contract would have been about the same as just hiring the fbsd core team at salary.

AFAIK, currently, Netflix only uses FreeBSD for their CDN appliances; their user facing frontend and apis live (or lived) in AWS on Linux. I don't know what they were running on before they moved to the cloud.

I don't think they started doing streaming video until 2007 and they didn't start deploying CDN nodes until 2011 [1]. They started off with commercial CDNs for video distribution. I don't know what the linux vendor marketplace looked like in 2007-2011, but I'm sure it wasn't as niche as in 1997 when Netflix was founded. I think they may have been using Linux for production at the time that they decided to use FreeBSD for CDN appliances.

> Hiring foss devs and putting them under NDA for everything they write that doesn't get upstreamed is an excellent way to get nearly everything upstreamed aswell, and the cost of competitors then porting these merged upstream changes back down into their linux is not nothing, so this gives a competitive moat advantage.

I don't think Netflix is particularly interested in a software moat; or they wouldn't be public about what they do and how, and they wouldn't spend so much time upstreaming their code into FreeBSD. There's an argument to be made that upstreaming reduces their total effort, but that's less true the less often you merge. Apple almost never merges in upstream changes from FreeBSD back into mac os; so not bothering to upstream their changes saves them a lot of collaborative work at the cost of making an every 10 years process a little bit longer.

At WhatsApp, I don't think we ever had more than 10 active patches to FreeBSD, and they were almost all tiny; it wasn't a lot of effort to port those forward when we needed to, and certainly less effort than sending and following up on getting changes upstream. We did get a few things upstreamed though (nothing terribly significant IMHO; I can remember some networking things like fixing a syncookie bug that got accidentally introduced and tweaking the response for icmp needs frag to not respond when the requested mtu was at or bigger than the current value; nothing groundbreaking like async sendfile or kTLS).

[1] https://web.archive.org/web/20121021050251/https://signup.ne...

The only patch to FreeBSD I maintained was hacking up FreeBSD 2.x to embed Hughes mSQL to back getpwnam etc and radius/tacass. I also needed to exceed the uid limit and came up with hacky uid for the customer and gid as the account letting each customer have the 30k or whatever account ids. I shared the idea with friends at Teleport ISP in Portland who ended up doing something similar with Oracle I think. I think the same sort of thing lead to vservers that best.net used with BSDi and eventually FreeBSD jails was developed and significantly better than my hack job so I migrated the isp over to using jails.

They're at the cutting edge. They're going to have to finance the development of most of this stuff anyway.

This development, but not all of the other things that go into a kernel and distro. If they pocket the whole core team, then they end up paying for all of the work, not just the few optimizations they really care about.

> In Linux land the cost of development is shared across many, which one might expect to yield a generally better product.

And you have more cooks in the kitchen tweaking things that you also want to work on, so there's a higher chance of simply conflicts, but also folks that want to go in a completely different direction technically/philosophically.

Are there papers out there from other companies that detail what performance levels have been achieved using Linux in a similar device to the Netflix OCA? Maybe they just use two devices that have 90% of the performance?