back
47 comments
Hey all, one of the authors here. I wanted to provide some more context on the questions asked here.

- Most of the tech described in the article is something you will have to do whether you choose gRPC or websockets. Most of the tech was about sharding and maintaining sticky connections and load balancing those connections across servers based on load.

- The heartbeat introduced on top of HTTP was needed to detect the broken connection faster and recover quickly as some of the payloads were very latency sensitive. Note that these connections are not 1:1 connections but have multiple hops across low bandwidth mobile networks.

- At the time we developed this initially in 2014, most of the WebSockets libraries would fallback to longpolling when the network connections were unstable. We explicitly went away from long polling. Since server->mobile payloads are the larger of the volumes, we settled in on SSE. Netty and the netty libraries provided most of the implementations of SSE out of the box.

At a certain mobile security shop ;) there was a push service rollout, back in 2013, and it was basically pings over the network, and mailboxes in Cassandra, no Redis layer atop. The volume was much smaller (~1hz), so it's interesting to see how the ecosystem has evolved, and how the volume requirements change the shape of the system
Good times!
On iOS, what happens if the user declines to allow notifications? Can you send silent notifications even when the user declines permission?
Yes except when user disables 'background app refresh'. https://stackoverflow.com/questions/30644343/is-silent-remot...
This push is not the native push like FCM/APNS that show notifications to the users. These are payloads that update the app (e.g., location of the cars on the map or eta) and never show any notifications to the user.
> This first generation of RAMEN server was written in Node.js using Uber’s in-house consistent hashing/sharding framework called “Ringpop.”

> ...

> Additionally, Node.js workers were single threaded and would have elevated levels of event loop lag resulting in a further delay in the convergence of membership information. These issues could result in topology information that is inconsistent and lead to message loss, timeouts, and errors.

>

> In early 2017, we decided that it was time for a reboot of the server implementation of RAMEN protocol to continue to scale. For this iteration we used the following technologies: Netty, Apache Zookeeper, Apache Helix, Redis and Apache Cassandra.

Yet another example of why starting with Java, .NET or similar stacks avoids rewrites.

I don't get the point of this article. What they achieved with the RAMEN service can easily be achieved by anyone using gRPC. At the very end of the article they even talk of a version 2 of their service which uses gRPC and seems promising.

Realtime push technically exists in every chat app on your mobile phone, so it's not a great technical feat either. Am I missing something?

This was published today, but these technical decisions appear to be as old as 2014 and there were other considerations that madan commented on here. gRPC was definitely far from mature up until like 2017 or so.
Is it just me or they have overestimated the problem?

For Uber’s use case I don’t see a reason to build the system that requires global scale. Users are often residing in the same geolocation. Why build this massive system that works for all instead of building smaller decentralized edge systems that does slower syncs in the back?

Very nice read nonetheless.

Resume-driven development?
Ramen v3 prediction:

- we separated payload from notification, now client is pulling payload on notification.

as someone that has complained about a lot of uber's resume driven development (see comments), I can attest that this system is

a.) necessary, and saves the company millions

b.) a simplification over the ... unusually ... architected system that came before and did not scale.

there's a lot of junk technology at many large tech companies, but scaling problems /are/ real when you have hundreds of millions of users and have demanding performance and reliability requirements. RAMEN is a differentiated and necessary part of Uber's infrastructure, even if that can't be said of a lot of uber eng blog posts.

What would be the second best way to do cover this use case with minimal effort instead of doing it from scratch? For lets say Amsterdam City Sized cab hailing?
I didn’t see them mention why they didn’t go with the more obvious websocket for the actual last mile communication with the device, since they wanted it to be bidirectional.

My guess is because doing this in Java is a huge pain whereas it’s practically a breeze in something like elixir.

Using websockets with Java surely is painless, for instance with Vert.x
> We have operated this system with more than 1.5M concurrent connections and pushes over 250,000 messages per second.

1.5 million? I thought Uber would have more simultaneous active clients than that.

The Google play store says Uber has 500 million downloads. Assuming ios has a similar amount, that puts the total number of downloads around 1 billion.

1.5 million concurrent connections would then represent roughly .15% of their total user base[1]. Hard to say whether that's accurate or not. I certainly use Uber less than .15% of the time but I imagine big events like New Year's see huge spikes in usage.

[1]Number of downloads isn't totally accurate for estimating user base but I'm not sure of another way. Also, article mentions that their system powers over 10 different apps but I'm assuming that the those numbers are negligible compared to the main Uber app.

Instead of switching all the way to gRPC for a return channel they could also have opened another persistent HTTP/1.1 connection which can be ready to send whatever is required.

gRPC (or more: HTTP/2) will win if the amount of request-levle concurrency that is required is extremely high. But in that case also the danger of head-of-line blocking between the various HTTP/2 streams gets higher.

'store network connections in zookeeper' is actually a compelling value prop for socket infrastructure once all the glue is resolved

wonder if this 'fireball' tool is really a general way to map realtime messages onto schema updates -- if yes, could potentially get uptake on other teams (if their schema format is at all portable)

Uber does seem to reinvent everything for itself - I’d be curious what their ops engineering spend is.

I’ve not fully understood how they can lose so much money given their fee per ride is so much higher than most apps would dream of for revenue

Uber spends a ton on side-projects that may one day bear fruit (driverless cars, trucking, etc) and stock-based compensation is a huge expense. I suspect their engineering expense for their core services is a small part of their losses.
Uber confirmed the sale of their self-driving unit (Uber ATG) to Aurora [0] a few weeks ago, and not everyone was absorbed into the company as part of the sale. So they're definitely cutting down on these "side-projects" to focus on the core parts of their business now that they have an obligation to shareholders.

[0] https://techcrunch.com/2020/12/07/uber-sells-self-driving-un...

Self driving(that you can trust) always seamed a like a fantasy. Last time I heard they sold that business unit/project.
Wouldn't want to miss that boat if it comes though. I think we'll have level 5 self driving for at least all purpose built roads within the next few years. I can't reconcile what Tesla have already released with not having self driving go mainstream soon. I suspect self-driving will never have a eureka moment either and will just slowly creep into existence.
The article states: >"These polling APIs ultimately just became a set of payload sharded APIs for the app to poll its features."

I'm curious would this essentially be sharding on HTTP headers that indicate payload type? Can anyone say?

Genuinely curious coz I don’t understand mobile networks very well — why not just send frames over websockets? You still get a persistent TCP connection and there’s no need for pushing or polling of any kind.
An open-source project (from myself) doing the same with WebSockets: https://github.com/Cretezy/dSock
+1 for integrating JWT as a first class citizen. Using JWT with websockets and therefore query strings is a lot of hassle with other frameworks.
So much effort to do UDP over HTTP ...

No wonder they need 1000s of SWEs.

So much effort to do UDP over HTTP

UDP on mobile networks is a royal pain because of symmetric NATs [1]

[1] https://en.m.wikipedia.org/wiki/Network_address_translation#...

I am not that much into mobile. Does TCP have the same difficulty?
TCP, especially on well known ports 80 and 443 is almost definitely going to work, if the network is going to allow any traffic. UDP might be blocked, and there's no well known ports more likely to work than others.

Other than that, NAT timeouts for UDP are probably a lot lower than TCP, so you likely need to ping more often to keep things active (but you should track this by network and adjust accordingly)

No, as long as the connection is initiated by the mobile device, TCP is much easier.
You saw their network reliability requirement right?
Ye, "Poor connection stability".

I don't see TCP/HTTP as a reliable protocol over unreliable networks. I mean they are sending hearbeats over HTTP. It is silly.

Why reinvent the wheel instead of using the state of the art RSocket protocol? Plus it handle backpressure for free, being reactive.
Recently I researched technologies for "real time web", events , reactive frameworks but this is now the first time I came across RSocket.

After reading the FAQ I am still not sure about its value propositions. Also I don't understand how "streaming responses and push" is different from the classic pub/sub model that so many websocket frameworks are offering.

I tried to avoid socket.io partially due to irrational preconceptions but then learned that it has a lot of features that are needed in many applications. It's also used by many people and has integrations into virtually every server and client framework.

Likewise, we could ask why Netflix reinvented the wheel, when they could have used the state of the art zeromq library.
When I considered a real time communication protocol for our app I remember ruling out zeromq but I don't remember why... Maybe they don't offer an API as clean/expressive? https://github.com/rsocket/rsocket/blob/master/Motivations.m... Or was it backpressure? I wonder what are the pros of zeromq vs rsocket
Real Time. Over the internet. What.

I actually looked it up.

https://en.wikipedia.org/wiki/Real-time_web

It feels like something a SW company wrote up for the sole purpose of making up the idea.

The "Difference from real-time computing" is uncited. Hell the entire thing is uncited except for Real Time search, from Google.

Are you complaining about some shitty Wikipedia article because an Uber article uses the same term "Real-time"? How is that related?
The problem is that "real time" has a meaning. A real time system has bounded latency and can make guarantees about how long something will take. See stream reservation [1] as an example.

The Uber system is not real time. It's entirely possible that every switch and router between you and the Uber server has it's buffers full. Round trip time is entirely unbounded over the internet. You could say their system is "fast" or "responsive", but "real time" is not a marketing buzz word, it has technical meaning.

IDK why OP chose to link to that weird article to try and illuminate this point.

[1] https://en.m.wikipedia.org/wiki/Stream_Reservation_Protocol

I think it’s probably not worth having this debate because nowadays, unless you’re actually talking about a valid context (like a rocket engine), the use of the term more often means “soft realtime” which is, “pushed out across systems when it occurs, not as part of a periodic batch process”
That's the common meaning of "real time" in embedded system land, correct. In web land, it tends to mean "if a user makes a change here, it auto updates on some other user's screen there without either user clicking 'save' or 'refresh' or something like that". Both are clear definitions in widespread use, and it's obvious that the Uber people mean the latter.

Your comment is like complaining that people writing a cryptocurrency blog use the word "crypto" wrong because obviously it means cryptography.

I think it's obvious that "real-time" has different meanings in different contexts. Besides operating systems we also have the "Real-Time Transport Protocol" (RTP) which has different timing constraints than an real-time OS.
Does knowing that your driver is about to collect you not have a bounded latency?

Given the necessary UX and the fact that Uber charges you if you don’t turn up for the ride, I’d say this absolutely has a bounded latency defined in their charging policy.

Real-time systems can have any bounded latency, it doesn’t have to be “fast”. I’d say Uber probably have a few hard-real-time systems around 5 minutes of latency, and a lot of soft-real-time systems around the latency of either 15-30s or around the duration of an Uber ride.