Building Uber’s Go Monorepo with Bazel

> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake.

But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that).

Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.

> There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

I think this is something that's slowly, but surely, changing. There's a lot of adoption of Bazel from some big players out there (Uber, Bloomberg, etc). The Blaze team @ Google has also started incorporating outside OSS maintainers into their OSS software (specifically rules_python).

Things are improving. Many of the things you've addressed (third party libraries, cross compiling, dynamically linking things) have been talked about by many people in Bazel's slack. It might not be flawless currently but Bazel's abstractions of how builds work are cool, RBE/caching is amazing for large projects, and a single build system with the promise of gracefully handling "every" language is something our field has been needing for some time.

In 3 to 10 years I can imagine being able to start writing code and never think about which language some library I want to use is written in, which package manager and toolchain I need to use to build everything, how to package my code to be deployable in production, how to setup unit tests, how to get autocompltions and docs and everything in my IDE, etc. The answer could just be "bazel, a language server, an IDE that speaks lsp".

CMake is a natural fit for C/C++ (even though I strongly dislike the syntax). It has a good IDE project generation, and you can use ninja to build super fast. But it isn’t pre-configured for hermetic builds out of the box, remote, team-wide builds and caching are much further than a command line option away.

I think the biggest thing to remember is that changing your build tool for a large project is going to be a massive effort, you're gonna need a LOT of hours to throw at it, and you might not find many benefits for a long time.

Starting a new project with it, so it can help you do things the "right way" is a much more attractive proposition.

That said, the IDE/tooling situation for Bazel was disappointing to me in the Java world too. It definitely lags and is incomplete.

I was keeping tabs on the Python plugin for a couple of years (but stopped checking in about a year ago) and Python3 was advertised as "supported" the whole time, but no one was able to get it working. There were a number of open issues, none with any action on them.

And while Nix has an insane learning curve, figuring out how to extend Bazel (and consequently build one's own Python plugin) was a nightmare. The documentation was unhelpful/incomplete and the codebase was incomprehensible and poorly organized, with inheritance sprinkled on everything for its own sake (which is to say, par for the course for Java apps).

I know... "Don't look a gift horse in the mouth! / Don't complain about open source software!" Fair enough, but I don't have to use it, and I'm free to caution others not to as well.

FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.

Bazel has extensive support for cross compilation. You can find pre-made toolchains. Intefacing with third party libs is as easy as writing a BUILD file.

Honestly the main reason Bazel is useful in a monorepo configuration is simply because it’s designed for it. It supports cross-language dependencies and generated files, and is designed for large repositories with many nested targets. Go’s build system is fine, but things like Go generate vs genrule probably come into play. Bazel also offers target visibility, to protect targets from being depended on in unintended ways, and build isolation, to allow builds to be more predictable (and help enforce target visibility.) It also has a few convenience features for vendoring 3rd party libraries and handling their licenses.

Of course Bazel is far from perfect, in fact sometimes you may be better off running your own rules instead of the official ones in some cases, but IMO it makes a pretty decent build system for putting all of your code in one place.

On the other hand, it’s one of those Google things where if you haven’t seen it in action in a functional configuration it’s hard to explain why it’s actually nice. And sadly I feel unsatisfied with the Node.JS rules for example, which is probably how a lot of people will first experience Bazel (since I believe Angular supports Bazel this way.)

As bazel is not limited to Go it can be nicer when the Go programs or libraries are part of a more diverse system. For example, you can build a Go library, along with the rules to generate a protocol buffer package, along with the C library that the Go library interfaces with, along with some static assets that you want to compile into your program all from the same BUILD file, and are built from a single command. I think to do at least some of these things, if they are possible with the go tool at all, you would have to run 'go generate' first.

Bazel also come with a query tool that makes it easy to tell what other targets your package depends on. Go also has such a tool, but again it's limited to Go dependencies.

I believe Uber has been using bazel for awhile now, and probably were invested in it well before the Go module cache was a thing, so Bazel's build cache is an important optimization.

The Go tool builds Go on a package by package basis. It won't help you with anything else. As soon as you need to build anything that isn't Go, or you need to do some pre/post-flight work, or you need to push artifacts etc etc etc the Go tool can't help.

As the blog post notes, Makefiles are the usual weapon of choice for Go projects of any sufficient complexity, but these are quickly outgrown.

The problem with Go's builtin toolchain that I've run into is dealing with anything that's not Go. For example, compiling protobuffers into go code. Go won't help you version the proto compiler, the proto standard library, etc. The result is that if left to its own devices everyone on the project ends up generating a completely different proto. If you check in the generated files, this is only a problem when someone wants to update the protos, but when certain files are harder to edit than others, people start coming up with workarounds that involve not editing them -- and the workarounds are scary. Any project where I use protos, I use Bazel because while it's much harder to get going, you are pretty much guaranteed that anyone who checks out the project and makes a change will get a binary with that change.

That said, I do agree with other posters that Bazel + go feels like nobody's priority. I use gopls and love it very much, but it doesn't understand how to ask Bazel where files are to analyze. A bug has been open for years, lots of people saying "I'll put it on my OKRs this quarter", and it's just not getting done. I am close to just fixing it myself.

But I have a feeling that this happens to everyone. I have worked on other Bazel projects and they all have detailed setup instructions; for example, Envoy uses Bazel but somehow shells out to cmake, so you have to have the right version of cmake, the right version of clang, etc. to produce a build. To me that defeats the purpose of using Bazel; Bazel should build the toolchain and all dependencies for you, otherwise what's the point? All you should need is Bazel and the source code. But that is not how people are using it. I don't know what the canonical successful Bazel project is, but I haven't found one yet.

I am currently working on a Typescript/Go project with gRPC/Web and will see if Bazel gets me that "one command full build" dream. I am somewhat motivated to make it work, but I have a feeling that there will be warts that makes me regret it. I used Bazel (well, the internal version) when I was at Google and every time I used it it was a joy. Incremental builds where you edited one file and wanted to run the tests were very fast, and builds where you modified some core library and wanted to see the impact across all the affected projects were also fast (easy to use 100s of CPU hours in just a few minutes of wall clock time).

Ultimately I think Bazel is the right approach, but I think to be successful you have to commit to having one engineer work on tooling full time. At your startup, probably not feasible. At Uber, probably very feasible.

(Going off topic, I would love to see what other people are doing for polyglot projects, because I feel like everything is a polyglot project these days. I have never seen a setup I liked -- install one program, check out the code, have a guaranteed working build that is bit-identical to the production image. The fact that nobody does this scares me deeply. But that's where we seem to be as a field.

I am also interested in seeing people's templates for projects. For example, I would like to do a C++ microcontroller project, and want to see how people are building those. I need clangd support, I need to use libraries, and everything I've seen is "hack together a makefile and cross your fingers". It just scares me.)

Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story:

Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using.

It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure.

The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell.

Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.

My experience working at Facebook was that the "working at scale" is nowhere near as glamorous as it sounds, especially when it comes to scaling build systems and monorepos.

As a company, you have to invest immense resources in evolving the infrastructure and it is a never-ending task: the more capacity you build, the more your organization will grow in ways that tax that capacity. The goal posts are ever-receding.

As an individual developer, all that fancy distributed infrastructure becomes a barrier that must be laboriously overcome at every step. Your expectations about what constitutes a "fast" operation get gradually distorted. Working on "small" side-projects (a few thousand files or less) feels effortless in comparison.

Not to say you shouldn't seek out the "big tech" experience; just that it is not all roses.

Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings

Uber Engineering sounds like they are collecting Google's open source tools and name-dropping them in their blogs to market themselves as "cool tech".

One companies boring tool is another companies favourite hyped third-party tech tool. (Until it doesn't work for them)

Scale is not a prerequisite, but you do have to sell it to your company. If not a greenfield, porting a codebase over is not easy, and you have to do your organizational excellence homework too. Monorepo, well reviewed code, team not in permanent crunch, high-quality tooling culture.

Just like Kubernetes, Bazel is not a silver bullet. But if you get all your ducks in a row, switching over is a gamechanger and will help you out even at the smallest scale.

You want to join Uber because you think Bazel is cool tech?

Having spent the last few weeks of my life converting our build system to Bazel I guarantee you it's far from what I would consider cool. It's more of a necessary evil because the alternatives are even worse.

Not just large companies - plenty of small startups building cool developer tooling. And much more fun when you get to start on a green field, rather than slowly migrating the existing code over the course of a decade.

They just had a laid-off recently, and still not profitable given how much $$$ they have burnt over the years. I am not sure if you wanna work there.

FANG such as FB and Google has some big monorepos, and you can try that if you are interested.

More politics, then tech.

I wish they weren't morally bankrupt as I would have loved to work on their tech stack.

The version of Bazel we (Google) uses internally has a lot of features that are not out of the box in open source Bazel. Mostly around distributed compilation or object caching.

Point is that take two sufficiently large monorepos & they'll have enough differences between them to require custom work

That Google had to come up with Bazel for themselves only supports that point. It isn't obvious whether it'd then be applicable to other existing cases without customization

Lots of knobs that vary here tho, such as where one draws the line between configuration & customization

go mod vendor && bazel run gazelle

No go_repository duplication.

>Monorepos are not efficient. They are easier to manage when a team is small but as the team grows and you have more and more deliverables with separate versioning you are introducing control structures in your automation. Complexity explodes!

The complexity exists whether it's a monorepo or many separate repos. A monorepo lets you encode that complexity as versioned code in your build system. Separate repos encode it across people's heads, wikis and who knows what else. Hiding complexity doesn't mean it doesn't exist, just that it will bite you 10 times as hard eventually.

That's a "holding it wrong" kind of affair, isn't it? Who expects to be able to "bazel test `shell thing with more than 4MB of output`"? Wouldn't a judicious application of either `...` or `xargs` fix this?

Google is a monorepo as far as I know, and they are doing fine.

But the number of targets is a function of _their_ code, not of using a monorepo, no?

If you need to build N libraries and M executables then you are going to have N+M targets (assuming building for one arch only) whether you use a monorepo or not... Either way you are going to issue N+M build commands.

Also, in bazel you can do

  bazel build //...

to do a full build of all targets in a workspace. If they are not doing this but instead passing each target name individually then that probably means that they are only building a subset of everything, and even that subset is too much to pass in a single command line invocation. I'll grant you that it seems excessive to have that many targets, but again I don't see having so many targets as an explicit issue of the monorepo.

A monorepo just lets you have commits across projects. It says nothing about how you build things.