back
118 comments
I'm using the sonnet at work. Work paid for it. Yes it's more expensive. I have gotten over 25 gbps bidirectional, but not by much. Maybe 27. A big downside is that it only supports 15W upstream power which I didn't think to check, so since that laptop only has 2 usb-c ports for power input, it's a bit limiting.

The upside, though, was that I plugged it in, got my throughput, and never had to worry ever again. Remember that engineering time is mucho $$. So troubleshooting overheating, divising a solution, manufacturing it, etc, that all eats into the savings.

The $400 dock would be cool though. It seems smaller than the sonnet and fixes my charging problem. Weirdly I didn't think I'd care about not including transceivers but they did actually come in handy, so that is a downside. Might try to get one of these if I need another one in the future.

>I'm using the sonnet at work.

I initially read this as SONET[1] and was seriously questioning your technical choices.

[1] https://en.wikipedia.org/wiki/Synchronous_optical_networking

Asking out of curiosity because I always wanted but never needed such speeds, could you elaborate on what you do and how does this speed help you? And I presume there’s a reason you’re not able to connect directly to the drives etc. with thunderbolt.
Can't you just put a real PCIe NIC in an eGPU enclosure with a fan? I bet you can solve this problem for like $150 and two craigslist transactions.
Thats what I do.

ConnectX-4 and newer are natively supported. I use a ConnectX-4 LX in a $50 Razer CoreX off of craigslist. Works fine, hit similar limits to what Jeff did.

Using an OCP is the neat trick here. MUCH smaller setup.

As far as the other posters comment about Tx's... that is an entirely different topic. I've never had an issue with FS.com optics personally.

Absolutely not. It's another layer of throughput/compatability bingo on top of the already often quite annoying SFP bingo.

If you are unfamiliar, the reality of the situation is quite bad. Slight hardware revisions of adapters/switches, which if you are lucky can be differentiated by manufacture date or serial number sequence, will not recognize known good modules, and if they do you then have to actually test if they can establish link, then let them burn for 24+ hours to see if they can maintain it without dropping (often, because of forward error correction, you will not start to see flapping until enough correctable errors have accumulated and the buffer overflows). Then there are firmware updates where this process must then be repeated.

You would think with how expensive all of this shit is you would simply be able to buy the officially listed as compatible components but my experience with them has been that a shotgun approach is best for your sanity and deadlines, buying sample kits of each module and trying them all, and then once you find the ones that play nice, buying as many of them as is reasonable and testing all of them.

If I had a dollar every time two modules from the same brand, that came in the same box, from the same production run, did not work with one another, I would be able to test your theory for free. Network Engineering is Fun

He mentions this in the footnote:

> I have SMB multichannel enabled, but this real-world speed limitation could also be related to my use of my lower-power Arm NAS (running on Ampere Altra, with 32 fairly-slow CPU cores).

Based on my own testing with a 25 Gb NIC setup on macOS I strongly suspect the issue is not on the NAS side, and the hardware I was working with is by no means underpowered.

The Mac I was working with is one of those rack mount M2 Mac Pros with a 25 Gb PCI-e card. On the NAS side it was a several hundred terabyte all NVMe flash multi-node cluster, each node with Xeon CPUs and 2x 25 Gb bonded NICs. Doing some rough throughput testing from macOS over multi-channel SMB yields throughput rates that are barely faster than the built-in 10 Gb copper NICs on the Mac.

Using iperf I can confirm that I can get near enough to 25 Gb/s raw network throughput on the Mac. And other non-macOS systems see significantly faster throughput to the same NAS. I am not sure exactly what the issue is that causes 25 Gb NICs to underperform so significantly for file sharing tasks, but I am reasonably certain that the issue is in macOS.

I’m in the process of gradually upgrading an in-house video production studio with <10 seats from 10G to 25/100G. There were attempts to do this years ago, and I found some unused Atto 40G Thunderlink adapters and Sonnet’s 25G adapters. The 40G Attos are EOL, and I couldn’t get the Sonnets to work. The SFP jungle/support/information is really maddening.

Atto’s new 100G Thunderlinks work well. I’ve seen speeds of just over 5 gigabytes per second. Atto definitely does a lot of kernel extension wizardry to get macOS to achieve these speeds.

A bit OT: I hope Blackmagic really shakes up the 100G connectivity market, and that Apple realizes their platform is extremely well-suited for IP Video 2110 workflows — if they can ever get networking under control.

One of the YouTube comments pointed out the issue is most likely the lack of support for SMB Direct (RDMA) in macOS. Would be nice to repeat the test on a Windows/Linux laptop.
This is a ConnextX4 on a thunderbolt bridge. This is well trodden territory. Unless some hero of open source has come to bat recently, MacOS doesn't support RDMA with this card or over Ethernet at all for that matter. It's only supported over Thunderbolt.

His iperf3 results show that it's able to go full beans on a single 25g port. A cross platform SMB file copy is kind of a meaningless benchmark in this context.

IMO a pretty cool product would be a network switch providing native thunderbolt interfaces to multiple upstream hosts. That wouldn't magically fix this RDMA issue, though. This is just a separate thought. Seems like the kind of thing Mikrotik should make.

I have a 25GbE setup using Thunderbolt, kinda similar to the author's. Except that my fan is held together by a piece of string, not a 3D-printed enclosure.

I found that SMB on macOS is just bad. If I try to heavily load it (e.g. by doing mass conversions in my Audible library), it can cause stalls and disconnections.

I managed to find a set of parameters (in /etc/nsmbd.conf) that makes it work more-or-less reliably by trial-and-error: dir_cache_off=yes, file_ids_off=yes, notify_off=yes, mc_on=no.

It's still terrible, though. It's so bad that Linux in a Parallels VM works _better_.

https://developer.apple.com/documentation/technotes/tn3205-l...

MacOS has supported RDMA over thunderbolt since March 19th.

It’s crazy that the Ethernet chips consume more power (presumably - because of the heat) than the processor in the computer. How is driving 25Gb/s such a hot task compared to driving multiple 40Gb/s Thunderbolt ports?
As I understand it, it takes a bit of anger to get electromagnetism to wiggle that fast over copper. Imperfect transistors using their full operational envelope to control a sort of chaotic thing? Fields and stuff.
They’re not that bad, but even a couple watts of power with no way to dissipate is going to get hot enough to burn you. Forced air cooling over a heatsink is really effective. 15W under load is much lower than a high end desktop CPU that can pull 150-200W.

See also NVMe drives? They draw enough idle power to drain a laptop battery overnight, and will easily hit 60+C in a bad enclosure. That’s only a 10Gbps module. The USB4 enclosures are all heatsink.

It's distance. 25GbaseT is good for 30m+. Thunderbolt is good for 1m without active cables. Making the much more rugged signal is a much more intense job.

And your computer already speaks the PCIe and thunderbolt. It has that probably on CPU. Here we have to receive USB, convert it to Ethernet. On much less up to date process nodes than your CPU.

Third, until very recently for even 10Gbit it meant using a usb4<->pcie bridge chip which itself was hot, to provide a generic pcie connection, which then talked to a conventional network chip, so it was a double hop: USB <-> pcie <-> nic. I don't know if 25Gbit to USB direct convert is available yet but we've seen a wave of vastly cheaper 10Gbit adapters come out that are single chip USB<->nic, probably not pcie at all, that are much much cooler. 25Gbit is probably still not here yet, maybe?

Ethernet is isolated through transformers on the physical layer and they're always sinking power. Thunderbolt is just pushing current directly through.

With ethernet it's for lots of reasons but a big one is not creating ground loops between distant equipment. If you didn't isolate the network cable would be sinking potentially amps of current between distant devices and that just isn't sustainable. You don't have this problem attaching USB between two devices.

A Mac Studio is over 100 W while a 25G NIC is under 25 W (that's the power limit for a normal PCIe slot).
Maybe the NIC does a lot? In past experience, a whole Xeon CPU couldn't keep up with a 10gig NIC if it was sending/receiving min-size UDP packets. Even with Intel DPDK iirc.
> It’s crazy that the Ethernet chips consume more power (presumably - because of the heat)

The NIC in this setup was released 12 years ago. It's built on a very old process.

There are no shortcuts in networking. It's quite easy to spend a 2-3x+ premium on a switch or adapter capable of non blocking line rate on every port simultaneously only to then have it fall over to a fraction of that in the form of lots of tiny packets. This is why we have things like QoS, flow control, and ECMP
is 10W a lot of power? I absolutely don't believe the cpu is using less than that while pushing 25Gb out

you can look up the rated power use of a connectx4 card, they do a lot of offloaded computation, too

Something like 5 percent of AI datacenter usage is just optics.
Dang, here I was feeling slick with 10Gb between my MBP and Mac mini and NAS. That’s the thing, it could always be bigger / faster / more. It’s enough for my workflow, but it’s fun to see people push further.
As mentioned in the blog post's footnote, the bottleneck may just be on the NAS side:

> this real-world speed limitation could also be related to my use of my lower-power Arm NAS (running on Ampere Altra, with 32 fairly-slow CPU cores). I was testing writing to an array of fast enterprise NVMe SSDs, so they shouldn't be the bottleneck here.

He was only getting 1 Gigabyte of throughput out of the built in 10 Gigabit Ethernet connection and moving to the 25 Gigabit connection only made a marginal impact on throughput, after all.

You're an order of magnitude off there. The 10 Gigabit connection was providing ~ 1 Gigabyte per second of throughput.
Jeff, I’m curious whether you really would have needed the $1,000 version of the Sonnet TB5 PCIe chassis. Would this $400 one have sufficed? https://www.bhphotovideo.com/c/product/1932655-REG/sonnet_ec...
$400 is for the empty enclosure and $1,000 is for the enclosure with a NIC in it. Obviously it's still a ripoff because a 25G NIC does not cost $600.
Somewhat off-topic: has anyone implemented USB C based networking?

So for instance you have two or more machines that have USB 4 ports at 20 to 80 Gbps and you plug in the appropriate USB C cables to connect them.

What do you need to make the connections appear as a (point to point) part of a regular TCP/IP network? Seems much cheaper than Ethernet if you already have the ports.

Yup, totally doable. Of course, it’s a point to point link so if you want to build a whole cluster that way then you’ll need a lot of usb-c ports or you’ll need to make more of a mesh topology.

I believe MacOS supports it via the usual network control panel. I’ve used it for transferring data from an older laptop to a newer one. The cable is more expensive than fiber, last time I checked.

I used to have a thunderbolt ring network for a ceph backend on my k8s cluster. On Linux load the thunderbolt_net kernel module and they show up the same as any other Ethernet interfaces. Just directly connect them and configure the network settings on each end using your normal tools.

Latency was an order of magnitude higher than twisted pair but the throughput was greater than 10Gbps and I didn’t have to spend on 10GbE capable servers and network equipment. I vaguely remember that they had to be proper Thunderbolt not just USB-C but that could have changed it’s been quite a while.

Yes, there are a few standard(ish) mechanisms for doing this, like [RNDIS][1] and NCM. If you have two Thunderbolt-equipped Macs and connect their ports together, you can use Bonjour/zeroconf for network discovery between them.

[1]: https://en.wikipedia.org/wiki/RNDIS

With USB 4, very strictly speaking: It's iffy. USB 4 doesn't require hosts to be able to switch roles and talk to other hosts. It can support this, but support is optional.

But for all certified Thunderbolt 4 and 5 ports, this kind of networking support is a requirement. (It is optional with Thunderbolt 3.)

The rest is up to the operating system.

USB-4 is basically Thunderbolt right? Thunderbolt networking is pretty automatic on Macs at least: https://support.apple.com/en-gb/guide/mac-help/mchld53dd2f5/...
You're limited in your network topology because there is no USB Switch. This HN post used two USB4 connections to create a 3-node cluster: https://news.ycombinator.com/item?id=39003469
Your so-called appropriate USB C cable is just two USB-C to Ethernet adapters plus an Ethernet cable. It’s very common on laptops without native Ethernet ports. But in my experience USB isn’t a very good protocol; it generally uses way more CPU than plain PCIe Ethernet. This is especially the case if the chipset is using ECM rather than NCM.
For us <25Gbps plebes out there: For god's sake do not buy a USB-C RealTek RTL8156-equipped multi-gig ethernet dongle to use with your Mac. I went through 3, and they're all trash:

Plugable 2.5G USB-C to ethernet adapter (bought November 2025), SABRENT USB C to 2.5Gbps ethernet adapter (bought October 2024), UGREEN USB C 2.5G ethernet adapter (also October 2024). In my experience they immediately fall on their face in a speedtest and drop to a couple hundred Mbps, experience packet loss, etc. This is testing across a few modern Apple Silicon Macs.

I now have a WavLink WL-NWU341G 5Gbps dongle with the RTL-8157 that seems to work well, but I've only run a couple of tests-- I am away from my home setup and don't have wired ethernet run to all the rooms where I'm living.

If you do some Googling, there's some good information out there about it, but when I bought the bad ones, I couldn't find any information corroborating my experience.

I have the SABRENT USB-C to 2.5Gbps adapter as well (also bought late 2024) and can echo this sentiment. I had it connected to a M3 Pro MacBook Pro and could never get anywhere close to 2.5Gbps from it. I never found other complaints at the time either and was starting to blame poor wiring in my walls until I replaced it with Ubiquiti's 5GbE adapter (which I believe uses a Marvell controller) and that has been rock solid for me.
I've had a lot of luck with the Plugable 2.5G USB-C adapters, although on some ports they're incredibly unreliable -- I'm guessing maybe something about the power supply?

On a CalDigit TS3 dock, I've had one working off the rear Thunderbolt socket for over a year, but it fails in minutes if plugged into the USB-C one. I've never had issues with a CalDigit Elements (TB4) hub. I've had success plugging them directly into various Chromebooks. On one Windows laptop, they seemed to work on one USB-C port but not the other.

So... reliable once working, but getting there can be a pain.

YMMV - I have three RTL8156 2.5G dongles - one in my Synology NAS, and two in different spots where I usually sit down in a chair and plug in my M1 Pro or iPhone to charge (on USB-C charger hubs) and have never had any issues where they drop out from throughput - at once point I had it running full bore overnight. They're from Cable Matters (2025) and Planex (2024). The only issues I have is sometimes they just don't show up when I connect and I have to disconnect and reconnect. That could also just be an issue with USB-C hubs.
I wonder if similar results would come from just adding a second 10 Gb NIC. If his workload runs SMB multichannel at sufficient queue depth, the load should spread across the multiple paths. If the queue depth is always 1, SMB multichannel is not helping on the 25 Gb NIC either.
What NAS drives can support anywhere near 25 Gbps?
How does Jeff Geerling afford so many Mac Studios?
You can purchase much cheaper TB to pcie converts in china. You add some generic 12v power and you're done.
I predict the next generation of Studio’s will have a 25 Gbps upgrade option built-in.
> I can edit 4K video straight off my NAS over the network,

Nothing against Jeff Geerling and I love his YouTube channel and I'm glad he's one of us HNer but this crop of circular "I make YouTube vids about big tx speeds and big NAS because I make vids hence I need big tx speeds and big NAS so I can speak about big tx speeds and big NAS because I make YouTube vids about big tx speeds and big NAS" videos is a bit tiring IMO.

I'm not into cryptocurrencies but, I don't know, host a full Ethereum node, host a Plex/Jellyfin server serving thousands of movies, host LLMs, etc.

I mean: talk about something else than editing vids because you make vids about vids.

What are typical use cases for needing 25Gbps?

Author talks about 4k video, but AI tells me that for "4K production mastering (ProRes 422 HQ, 4:2:2, 10-bit)" the rate is under 2Gbps.

I'm not saying it's not needed, - just curious.