▲ 376 points
back
105 comments
Many people don't understand how or why the linux kernel is the bottlebeck at speed higher than 10 Gbps.
The problem is that the linux kernel can't process many simultaneous small connections.
Just to be clear: - linux can easily transfer at 40 Gbps - linux chokes at around 1M packet per second per cpu socket.
Thats right.
So linux can easily transfer at 40 Gbps one or few simultaneous flows
But! It can't transfer several flows at 40 Gbps.
The bottlebeck is the number of packets per seconds it can inspect.
So if you are sending 100 big transfers, you will reach 40 Gbps.
If you send 5M small transfers, linux will die.
This is why netmap is handy.
It offloads the packets from linux directly to the app
Bbc is sending xM small requests per second, hence millions packets per seconds.
In conclusion, netmap is good if you need a lot of small simultaneous connections.
I reached 40M connections on chelsio 40 Gbps nic using netmap on FreeBSD 11.
> linux chokes at around 1M packet per second per cpu socket.
which is why kernel bypass mechanisms f.e. dpdk/netmap/... etc. are getting some/lots of traction. fwiw, using dpdk (on x86-64 hw), i have had no problems with pushing minimal 64b sized packets for large (>250k) number of flows at line rate...
This isn't specific to Linux. PPS is a limiting factor for network hardware vendor's gear as well. Lots of ~100 byte packets will bring pain to Juniper and Cisco products as well.
Thanks, I've always wondered under which conditions BSD networking performs better compared to Linux. It's the number of simultaneous streams as opposed to raw bandwidth for fewer streams.
Well BSD kernel will also choke at more or less the same limits as linux kernel.
The key point here is that a netmap driver bypass the kernel and therefore open the door to many millions packets per seconds or millions requests per seconds. Not many Gbps
Next time management asks why the driver for your product should be open source, point out that it makes your product much more attractive to people who will improve and promote your product for free.
That is, sales will improve.
Sales improve because the potential userbase has increased, not because of free contributions.
Imagine you are google and you have created your own custom build system. You now want to hire more people that are familiar with it. Unfortunately the potential applicant pool is exactly 0 because nobody outside of google can even start to learn how to use the google specific tools because they simply don't exist outside of google.
The big barriers to adoption of a new product, service, or whatever are:
1. People knowing about it.
2. People needing it.
3. People trusting it.
Like a fire needs heat, oxygen and fuel, new products survive only when they get all three corners of the triangle. If your product is cheap enough, some people will try it when they think that they have a need for something like that; trust comes eventually. Open source builds trust that in the worst case, people using your product will have a glide path out instead of a sudden brick wall.
In the instant case, Mellanox gets a highly visible, reputable customer telling everyone in their industry that Mellanox NICs are high-performance, are trusted, and can be adapted to their needs. Anyone who reads this article and thinks about high-performance NICs will have a bit more trust in the Mellanox brand as a flexible system.
if you meant Mellanox, this is exactly what i thought
I vaguely remember similar story for FreeBSD from Netflix or Limelight. And it was Mellanox as well.
> it makes your product much more attractive to people who will improve and promote your product for free
This is a huge myth.
1) even if you open-source, the company retains control over the project, and push their priorities first.
2) codebase can be huge and intricate, very few external contributors would actually put the time to learn it, leading to poor contribution (ie. might fix a given problem, but breaks other subsystems)
3) even if there is external contributors, is the company willing to put the manpower to process the changes and handle community interaction ? Ie. do you really want to handle bikeshed'ing style patches, potentially costing time to actual paid devs ?
Most of the time, niche opensource contributions are merely are more of a marketing tool (or HR tool ala. "your profile looks great however we're too greedy to hire you, please contribute to our software for free").
It's a good thing for the BBC that Mellanox didn't agree with you, eh?
#1 is trivially true for all products.
#2 is essentially saying "if your code is crap, you will get crappy contributions". Well, if your code is crap, maybe you want to fix that?
#3 is valid. You don't necessarily get great results without putting in a little effort.
Lots of people missing the point in this discussion, the goal of this is to send Uncompressed video over closed IP networks in a studio for example. I talked about some of the problems at Demuxed last year (referencing BBC R&D's work): https://www.youtube.com/watch?v=A4L5xEXXlas
I don't think many people missing this point. I got the point from the article itself and even before this when I saw BBC I figured it was going to be related to video.
One of the sibling articles explains that part of the project involves trying to cut labor costs on covering massively multi-sited events:
https://www.bbc.co.uk/rd/projects/nearly-live-production
The idea is explicitly that you would run the equivalent of a TV control room on a web app that knows how to switch between video feeds (potentially from fixed cameras without human operators) and then can output video to... whoever is your audience.
The "moving window" of near-live editorial decisions is pretty interesting as well, and seems geared towards giving non-professional editors a chance to fix mistakes.
How does a web app know how to switch between video feeds, or cameras know where to point?
Maybe we're talking about different kinds of events, but there's actually a lot of creativity that goes into camera framing, movement, shot selection, timing of cuts, etc. to end up with something that's actually interesting to watch...
relevant TLDR:
"Professional, live, multi-camera coverage isn’t practical for all events or venues at a large festival. For our research at Edinburgh Fringe Festival 2015, we experimented with placing three unmanned, static, ultra high definition cameras and two unmanned, static, high definition GoPro cameras around the circumference of the BBC venue. A lightweight video capture rig of this kind, delivering images to a cloud system, could allow a director to crop and cut between these shots in software, over the web and produce good quality coverage ‘nearly live’ at reduced cost."
> unique challenge here involves handling IP packets (around 1500 bytes each) at data rates of between 1 and 8 Gigabits per second
This reminded me of something. At this point it seems like Jumbo frames are never going to be widely adopted, are they? Otherwise this seems like the perfect application - massive datarates, controlled hardware/software, high-quality wiring...
Working with Mellanox cards for a few weeks; the MTU matters a lot. The difference between an iperf test measuring 14 Gbits/s on MTU 1500 vs. 40 Gbits/s on MTU 9000.
We got a couple of Connect-X5 cards, which allow switchless connections, akin to a ring topology. A lot of neat things, at just stupid line speeds, and latency levels I haven't seen in software, ever.
For my home LAN the problem with Jumbo packets is that it seems there isn't a standard size that all NICs support. I have several machines that support 9000 bytes, some 9014 bytes, some only up to 4000 bytes. So I haven't been able to get the whole network on a single MTU. At least this was a case a few years ago.
Jumbo frames are limited in the improvement over 1500 octet frames, around 9000 bytes. That's still too small. I think NIC offload mechanisms already handle bigger chunks of data and let the OS TX/RX 64k or more at a time.
BBC R&D gave a talk on this very project a couple of years ago at a UKNOF meeting, it was very interesting:
Last kernel hacking I had to do some work with 10Gbps NICs, so 80Gbps is just so attractive to my eyes.
There are different pass-through/fastpath patch for different chips to avoid memcpy, or do zero-copys, but they are all kernel patches, kernel-only.
An alternative method will be BPF/XDP and DPDK for which you will need modify the kernel drivers somehow for good performance. Wondering if Netmap does that already or is has nothing to do with them.
All of them are addressing Dataplane packet move, hope I can have an environment to experience these close-to-100bps network in my next projects.
In the meantime, I am wondering, why do you pass 4K uncompressed video using IP packets...
Wonder why they didn't use DPDK instead of rolling their own MLX specific thing.
DPDK is a very recent thing...
Can someone ELI5 the major challenges of delivering "broadcast quality" video over IP compared to cable? It seems crazy that pushing HD content over a standard aux cable is faster than downloading HD over IP. We have been pushing HD content to our TV's for 10+ years, but many people still have trouble streaming an HD movie from Netflix.
Is this simply due to the overhead of IP transport supporting bidirectional communication? That is, a TV broadcast only needs to support a fixed set of N unidirectional flows (channels), but IP needs to support a dynamic set of N bidirectional flows?
It's essentially the difference between circuit switching (dedicating a whole physical link to a particular signal) and packet switching (breaking it down into smaller chunks that can each be dealt with independently throughout a network, and then get reassembled on the other end). The former is easier to do (no overhead of disassembling/reassembling) but less flexible.
Challenges are mostly around keeping lock-step timing across a lot of separate devices in the system at scale (using PTP) for frame-accurate switching, and working through getting the SMPTE-2110 standards completed for interop, but there huge benefits on ability to scale to well beyond what is practical with coax, as well as cost savings on cabling with more density (and weight savings, for mobile production trucks which is important). There are also big architectural advantages of having more dynamic infrastructure and device discovery (AMWA NMOS, etc) rather than hard-wired coax signal paths.
There are a lot of scenarios with massive scale on the back-end, with lots of streams managed as part of the production process ahead of delivery to consumers... We have an IP video router (https://evertz.com/products/EXE-VSR) that has 2,304 10Gbps ports (moving to 25Gbps per port), each of which can do 6x fully uncompressed SMPTE 2110 video (1080i 29.97fps) flows in each direction, with a 46Tbps non-blocking back-plane so the scale can get a bit crazy.
The full scale back-end stuff is pretty invisible from the consumer side. Eventually that internal infrastructure feeds into distribution encoders that produce lower bitrate streams for cable/sat/web distribution (for real-time events, with separate file delivery for VOD platforms like iTunes & Netflix)
As you said, broadcast is usually done with IP multicast, so it's not really bandwidth intensive. The netflix example is different, there you have a large group of people who all want different streams. At netflix's scale serving that many simultaneous connections from any reasonable amount of datacenters has the capability to saturate DC uplinks at peak hours. Hence why they've resorted to caching appliances in IXs.
I would guess cable video is multicasted once to many customers while online streaming has to be served one-to-one.
From the viewer's standpoint, the bandwidth usage is roughly the same, but the broadcaster and CDNs have a lot more data to deliver.
There's a lot of orgs out there developing broadcast IP tech in house. Would be interesting to see an industry wide consortium on this
There are some consortiums that are starting to form: http://www.videoservicesforum.org/activity_groups/RIST_poste...
BBC R&D are participants in one: https://www.nmos.tv/about_NMI.html
So if this lossless video over ethernet thing ever trickles down to consumers, what happens?
Can my computer monitor become just another network device plugged into Ethernet like my printer already is?
In my living room, can I ditch the video routing part of my AV receiver and just plug everything (streaming devices, video games, cable box) into a very fast ethernet switch?
Quickest Ethernet is up to 400 gbps now, you can probably get by with a pedestrian 10gbit switch :)
Hard to see uncompressed consumer video catching on though since compression works so well.
That would be an enormous waste of bandwidth, considering that a fullHD stream (30fps @ 24bpp) would consume 1.4 gigabit/s.
am i reading this right? are they coming up with a way to use IP to broadcast "television" in the sense that, if this was finished and open source that "anyone" could feasibly run their own "television channel/station" (edit: over IP) ?
Not exactly. This is about how to pass video between interoperable machines.
Traditionally within a broadcast house, video+audio was sent between machines using HD-SDI using BNC coax[1]. For the first generation of HD, this ran at 1.5Gb/s. For 1080p/59.94Hz, a new standard was developed to run at 3Gb/s. For 4K at 59.94Hz, there is a 12Gb/s standard.
HD-SDI routers are extremely expensive. Each input and output from a device has to be individually cabled to a router and some devices can have many dozens of I/Os. There has been a push to switch from using ridiculous numbers of cables to using IP solutions and off-the-shelf IP routers. SMPTE 2110 provides a framework for doing this using RTP [2].
4K needs roughly 12Gb/s, so to carry a single video, you need at least 25Gb/s networking. If you want to carry many signals, the network bandwidth goes up fast -- that's why BBC is interested in 100Gb/s links.
Also keep in mind the traffic is not "bursty". These links may be nearly saturated 24/7. Most off-the-shelf routers are not designed with enough buffering to handle fully saturated links on all connectors.
[1] https://en.wikipedia.org/wiki/Serial_digital_interface
[2] https://en.wikipedia.org/wiki/SMPTE_2022 (older IP standard)
not necesarily to broadcast television, but to produce it. Rather than having to buy expensive bespoke hardware to fit a TV studio, what happens if it was just software running on a bunch of servers (maybe in a cloud)? This is the background of the project: http://www.bbc.co.uk/rd/projects/ip-studio
Also. live TV. Live feeds over IP have always had their issues. Slight delays, even the concept of buffering, is not acceptable. Packets have to come in and out of the system at roughly the speed of light. Any drop of anything will be seen on screens. Parity can help, but that requires processing that is very difficult in the acceptable timeframe. A typical TV studio could have 20 different live feeds coming in and four or five going out. They need to switch between and/or layer any number of these instantly and seamlessly. The bandwidth demands quickly become insane.
Remember too that the feed will transit over many different IP-based systems. A seemingly acceptable delay on one system, a few microseconds here and there, adds up or multiplies as data streams between equipment.
I think it's from the cameras and recording gear to the backend where it's peiced together and that is what is broadcast... given they mention uncompressed hd and uhd, that's the impression I get...
If this is for a more-or-less closed system, I wonder why it uses IP, and not an RDMA-type network (Infiniband etc.) which already is kernel-bypass and works routinely at around 100G, 1μs latency. There's nothing special about the Mellanox drivers amongst the Openfabrics ones in having free drivers in Linux, but they generally require blobs or separate firmware.
Wouldn't products like openvpn benefit from using this? Or am I missing something?
Not sure if that's a complaint or not :-)
My view is that of all my susbscriptions for entertainment - sky, disney, netflix, and time wasted listening to ads, the one that gives easily the best value for money is 12 quid a month to the BBC.
(I mean sky is nearly 60 quid a month and i cannot persuade my other half to go broadcast-less)
I never understood why they stick with this model instead of just levying a tax on citizens (like my beloved CBC). The BBC has been more than just a TV station for a long time; and I hear people in the UK are dead tired of having BBC license "collectors" coming round every 2 months to harass them into paying fees for the flatscreen TV they have hooked up to an XBox.
I will be curious to see how well the new Google kernel, Zircon, performs compared to Linux.
I have my doubts a micro kernel can come close to the Linux kernel efficiency.
Zircon is the new micro kernel that is part of Fuschia.
Isn't this a bit of an odd comment, on a story about kernel bypass for high performance networking?
Cool.