back
332 comments
This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens.

I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies.

(The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)

Former Uber engineer/EM here: I worked on the Rider app.

The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city.

The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or just not relevant to the type if user - like business traveler use cases).

Uber builds and experiments with custom features all the time. An experimental screen built for London, UK would be part of the app. Multiply this by the 40-50 product teams building various features and experiments outside the core flows you are talking about (which core flows are slightly different per region as well).

I worked on payments, and this is what screens and components are in the Uber app:

- Credit cards (yes, this is only a a few screens)

- Apple Pay / Google Pay on respective platforms

- PayPal (SDK)

- Venmo (SDK)

- PayTM (15+ screens)

- Special screens for India credit cards and 2FA, EU credit cards and SCA, Brazil combo cards and custom logic

- Cash (several touch points)

- AMEX rewards and other credit card rewards (several screens)

- Uber credits & top-ups (several screens)

- UPI SDK (India)

- We used to have Campus Cards (10 screens), Airtel Money (5), Alipay (a few more), Google Wallet (a few) and I other payment methods I forget about. All with native screens. Still with me? This was just payments. The part where most people assume “oh, it’s just a credit card screen”. Or people in India assume “oh it’s just UPI and PayTM”. Or people in Mexico “oh, it’s just cash”. And so on.

Then you have other features that have their own business logic and similar depths behind the scenes when you need to make them work for 60 countries: - Airport pickup (lots of specific rules per region)

- Scheduled rides

- Commmuter card functionality

- Product types (there are SO many of these with special UI, from disabled vehicles, vans, mass transport in a few regions etc)

- Uber for Business (LOTS of touchpoints)

- On-trip experience business logic

- Pickup special cases

- Safety toolkit (have you seen it? Very neat features!)

- Receipts

- Custom fraud features for certain regions

- Customer support flows

- Regional business logic: growth features for the like of India, Brazil and other regions.

- Uber Eats touchpoints

- Uber Family

- Jump / Lime integrations (you can get bikes / scooters through the app)

- Transit functionality (seen it?)

- A bunch of others I won’t know about.

Much of the app “bloat” has to do with how business logic and screens need to be bundled in the binary, even if they are for another region. E.g. the UPI and PayTM SDKs were part of the app, despite only being used for India. Uber Transit was in a city or two when it launched, but it also shipped worldwide.

And then you have the binary size bloat with Swift that OP takes about.

Follow-on tweets address this somewhat: Every airport has different pickup locations, every region has different payment rules including cash, etc.

Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.

I have to say, this comment is quite ridiculous. Might as well say that Google is simple because it's just an app with one search field.

I never work for Uber but I can easily imagine the following problems:

* Map data is not a solved problem. Tons of inaccurate data especially on non-english speaking countries.

* As someone mentioned about payments, different country requires different payment platform integrations.

* Payment is just one aspect of money, you need to build a treasure-like logic to run a marketplace between drivers and passengers.

* The matchmaking algorithm itself is interesting enough problem. It has to be efficient for the company to make the most amount of money.

* Tons of internal data are produced and required to be understood. This requires ML/data analytic talents.

There’s a lot of complexity behind making the user experience as simple as it is. One example: it’s my understanding that there is a lot of effort put into correcting for GPS error in areas like cities so the the little pickup location dot is in the right place. I imagine the entire system for that is a fair amount of offline processing, backend and client code.
From one of his replies on Twitter:

> The app looks simple from the user perspective. But on the global scale the business rules are insanely complicated. Every region has custom rules/regulations. Different products have different workflows. Some regions have cash payments. Every airport has different pick rules...

A negative / unpopular take: like air in a balloon a startup company will grow as large as its capital allows as it is purposefully using this capital to grow faster than natural market forces allow. In other words it has a big engineering team because it raised capital to get a big engineering team. Craigslist is a great example of bucking the trend in this regard.

A more positive take: if you look at the specifics of how many regions Uber is (was) in, and that each region has cities each of which have wildly different regulatory environments, you can imagine the permutations of complexity they had to deal with. Then you add in growing internal business experiments like Eats or mobility (Jump) and it's easy to see how this balloons out of control pretty fast.

Experimentation / data-driven product development. You’re actually downloading 583 small variations of the same app, plus all the localized rules mentioned above, and it quickly adds up.
>This may be an egregious question, but what does the Uber app actually do that makes it so big?

It justifies the employment of 400 iOS engineers, it also lets you book taxi rides.

I’ve often wondered the same thing about Airbnb, DoorDash, and other unicorns where they hire the best engineers from Silicon Valley, yet their product seems like a rather basic CRUD tool.

Apparently with Uber it’s the infrastructure design and back-end algorithms for trip routing and driver management that are “so great.” The SV hype machine helps a lot too.

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it.

Problem here with Swift came down to

- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)

- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)

Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.

Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.

Great thread.

> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.

> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs

This would scare me too.

> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.

This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.

I've lost track of the number of "biggest engineering disaster" stories I've read that begin "so we decided to rewrite it..."

The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.

Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point.

Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"

Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.

I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches.

I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations[0][1].

[0] http://read.pudn.com/downloads196/ebook/923902/ShowStopper!-...

[1] http://www.proudlyserving.com/archives/2007/09/dave_cutler_w...

If I could read a book of these stories I'd stop everything and read it now. Absolutely incredible. I love stories like these, but I also understand these hells are probably reserved for the special world of startups. The interesting takeaway from this for me is the question "Was the rewrite successful?".

As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me.

Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story.

The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discovered that our compiled code size was growing a rate of 1.3 mb a week ... But the real problem was the growth curve. It was always eating our winnings back.

This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible.

The larger community benefited from our learnings.

No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen.

So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.

No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.

> So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.

I really like this point.

It was pretty obvious when Swift 1.0 launched that it should’ve been called Swift 1.0 beta 1. Not a knock against anyone who worked on the project, other than senior management at Apple.

Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.

I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’

Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here.

Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.

I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it.

> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.

> With only a week left we decided eat the 8 figures and drop support for iOS 8.

I'll say this: If I were an Uber investor I would be pretty angry. And the story just gets worse from there.

This is a story of gross incompetence. It's not an engineering disaster, it's a management disaster. This is classic "inmates running the asylum". ( https://www.goodreads.com/book/show/44098.The_Inmates_Are_Ru... )

This is the story of some hackers who got in good at a big company with lots of money and sturm und drang so they had no adult supervision while they partied while pretending to be actual engineers. (That they were pretending to themselves too doesn't make it any better.)

The writing was on the wall, but rather than own up and take responsibility they double down and keep messing things up for months and years.

> This is when the real brilliant engineers started to shine.

This is when they get pulled from other tasks to clean up your mess! (cf. Broken window fallacy.) Those real brilliant engineers should be doing other better things than dark rituals to appease the Swift compiler gods.

Look, we all make mistakes. There's a blog post from an engineer at a company I used to work for complaining about the pointless middleware I wrote years earlier. (I feel ya bro, I told them at the time that it was pointless.) Here's the thing: I got out of that place when I realized that the folks running it weren't really competent. (And, sure enough, they got pushed out by the investors and now the company is a "Wallstreet darling". So... Yay?)

I've burned enough investor money in my career that I don't think it's funny anymore. (I never really did think it was funny, I don't like waste, but I was young. REDACTED was only my second official job, and in hindsight it was really crazy. I mean, they built a skating half-pipe in the office! At the time, everybody thought that was real cool.)

One company I worked for a few years ago, I joined right as a just such a ground-up rewrite of their core system was coming into place. Everyone in orientation was telling me "You'll be spending most of your working life dealing with <new project codename>" but they were actually training me on the existing system. About a month out of orientation there is a company wide. They canceled <new project codename> because it wasn't meeting it's technical specs.

It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.

This to me reads of Swift being pushed hard well before it was truely ready. Polynomial linker time? Wow. Ouch. No thank you.
Apple's botched introduction of Swift is easily responsible for billions of dollars in lost productivity across the industry since 2014.
In some other thread someone who regularly attended meetups hosted by Uber in NY said this (paraphrasing): "You could rely on Uber coming up with the most complicated solution to problems they create for themselves. But the food was good" Pretty much sums up anything I have ever read about uber engineering.
This story at Uber is a fascinating example of the growing pains involved with transitioning to a new language.

LinkedIn adopted Swift back in 1.2. I would love to read some war stories of how things there have progressed over the years.

Source: https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37)

At an interview with them a few years ago, I was told they had been moving towards rewriting the app in Objective-C.

Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.
Binary size should have been treated like a limited resource: set a hard limit in the test suite, and let the engineers who commit code in the organization fight it out how to delete unused code when they create new one for a new feature.

I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):

https://github.com/geohot/tinygrad

Here's the GPU ops part:

https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_...

Definitely thought this was going to be a story about when their car killed someone: https://www.nytimes.com/2018/03/19/technology/uber-driverles...
Were all of these.....tweets originally? Or does the author write in this crazy "few sentences per paragraph then break" style? So hard to read.
Great thread. Thanks for sharing!

It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that.

I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?

Computing is always susceptible to shiny object syndrome. Just because it’s new doesn’t make it better. I wonder if things would be different with a more nuanced approach to Swift adoption, knowing what they knew of its limitations already, and being realistic that the language is not a panacea (none are!)
I know C++ has its warts, but almost every mobile project I've worked on has been C++ with a thin layer of objective c and java interop. IMO it's really not worth it to get so tied up in a single companies tooling even if the developer experience is superficially nicer. C++ has all sorts of awful, but it's familiar awful with mature tooling.

Admittedly the things I've worked on has been games and CAD systems so the internal data models and simulation are a lot more complicated than the UI, but I think that holds for a lot of apps.

> Some of my ex-Twitter colleges called on me to quit such an evil company that would track you like this.

Twitter owns an company called MoPub that broadcasts user location information to the adtech industry.

The biggest engineering disaster Uber had was when it intentionally disabled the automatic safety brakes on their 'self driving' car and killed a person.
"Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience. So to improve location pickup we changed the location permission to collect signal in the background so we could send the drivers to your current location. People freaked out."

This doesn't make much sense. When you open the app to schedule a pickup, and you have location tracking enabled while you use the app, Uber should be apprised as to your current location within seconds. Where you were before you opened the app should not matter.

It would be useful if the author could clarify this.

As one of the people who started a petition to Apple to disallow developers from disabling location tracking only while using the app (i.e. all the time or never, which Uber tried to force at the time), I am partly responsible for the outcome. But from what I can tell, no customers were worse off as a result.

If Uber has data showing otherwise, that would be interesting.

My mind is constantly blown away that Uber app needs to be so big.
Am I the only one who wants to scream when encountering long-form content on Twitter?
Who's making the version of this post for Netflix?
Fun fact: I was able to access a bunch of employee admin screens in early 2015 because of the client side architecture of the app. I noticed around a year later the architecture had changed and always wondered if my article had a role in the transition

http://nathanmock.com/archives/how-i-accessed-employee-setti...

Wow. The real lesson here is that all this "engineering" and doing stuff the supposedly elegant and better way was a colossal disaster.
Seems like the lesson here is to not adopt new technologies on your flagship products (instead of first testing the waters on smaller products).
I’ve seen many rewrite stories go well. I’ve seen many go south. This just seems like awful management and not spending time early de-risking things.

That being said it’s a common problem in growth startup. Prioritization is hard. Too many engineers doesn’t mean problems get solved. They need to be able to talk to each other and execute with what’s essential to users.

Growth is a curse and a blessing.

Not a mobile developer -

Was the move to Swift inevitable and they started too early or should they have stayed with - and still be using - Objective-C?

I want to hear the Android side of this story!
Gone are the days where Twitter body limit was understood as the reason to post short tweets instead of long tales...
> As a consequence of all these problems, there began to be a growing movement across all levels of the org that was rallying around the idea of “rewriting the app from scratch” The general sentiment was that the architecture was slowing us down, starting over would be faster.

I've read plenty of these stories where a rewrite was a total disaster. (Most famously, Netscape.) I've seen and/or lived through some too. (One where a major distributed system was completely changing its storage model, and it was thought refactoring was impossible. In hindsight, it was achievable and would have saved _years_.)

Anyone have examples of regretting a refactoring to the same extent?