▲ 194 points
back
166 comments
Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precisely specified declaratively. The urge to hack in a little imperative code or jam in a build step that you just "know" needs to happen in a certain order is always there and it can be very difficult to satisfy the taskmaster that is the build system. When you have weird cases like a script that produces a variable set of output files based on the content of some input file, it's tough to get that working in the build system.
But... if you climb that cliff, there are many rewards at the top. Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state), you can automatically parallelize (because the build system can tell declaratively which tasks do no depend on each other) and you can ask it interesting queries about the dependency graph. That sounds an awful lot like the runtime safety, performance, and static analysis static types give you.
The challenge is that for small programs, dynamic types are just easier to get going with. And every build script starts pretty small. So you have a plethora of simple, hackable build systems like make that work fine at first but then get more and more painful as the program grows. With software, there is at least some customer-visible pain that can enable you to justify a rewrite in a statically-typed language. But build infrastructure is always a hidden cost so it's really hard to get the resources to migrate to a different build language.
Maybe the lesson here is to bite the bullet and start with a hermetic build system at first.
Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just:
cc_binary(
name = "app",
srcs = glob([ "*.hpp", "*.cpp" ]),
)
Yes but there is a hundred megabytes of system behind that. Tens of thousands of lines of starlark code, and tens (maybe hundreds) of thousands of lines of java code besides.
A lot of work goes into those 4 lines. This is evidenced by the simple question of, how do you set up a custom compiler for that code properly? The answer being a couple hundred lines of starlark code, for what in other build systems is effectively `CC=clang` (to be fair bazel allows this but does not support or encourage it, especially for anything besides toy builds).
I like Bazel (a lot!) but you are oversimplifying it here.
Simple things like this are simple in all C/C++ build systems, the question is how the complex things are handled (multi-platform support, detecting and selecting different versions of the same compiler, cross-compiling, IDE support, multi-stage builds where source files are generated in the build which need to be inputs to later build stages, custom build steps, or bootstrapping tools which need to be used as compilers or code-generators for later build stages, etc etc etc...)
The syntax is almost never the issue.
The issue is that declarative build systems ALWAYS need an escape hatch for the exceptions--and the exceptions grow over time.
We've been here before. It was called Make. And then it was called Ant. And then Maven. And Google still built their own system. And they will build another again.
Nobody ever learns the lesson that a build system is a program--period. Sorry to burst your bubble: there will be circular dependencies--you will have to deal with it. There will be two different C compilers on two different architectures. You will have to deal with it. Libraries will be in the wrong place--you will have to deal with it. These libraries come from the system but these libraries come from an artifact--you will have to deal with it.
The only way to deal with it is to have a genuine programming language underneath.
I guess but this is also portable and more people know how to extend it:
$ cat Makefile
SRCS = foo.cpp bar.cpp baz.cpp
OBJS = $(SRCS:.cpp=.o)
PROG = app
EXT =
.PHONEY = all clean
all : $(PROG)
LDLIBS = $(OBJS)
$(PROG) : $(PROG:$(EXT)=.cc) $(OBJS)
clean :
rm -f $(PROG) $(OBJS)
$ make SRCS="`echo *.cpp`" clean all
rm -f app a.o b.o c.o
c++ -O2 -pipe -c a.cpp -o a.o
c++ -O2 -pipe -c b.cpp -o b.o
c++ -O2 -pipe -c c.cpp -o c.o
c++ -O2 -pipe app.cc a.o b.o c.o -o app
$Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.
> Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state)
Note that Bazel builds still depend on the ambient system environment, like the compiler toolchain and system libraries, and it's possible to poison a shared build cache.
Shared caches are typically used with remote execution and CI servers, where the build environment is fully controlled.
Doesn't Bazel hash the compiler and env variables to prevent this?
The only way to poison the cache then is to break things at an OS level, which is highly unlikely.
Hermetic build systems make a lot of sense for companies, where you often want to control your whole stack and might even use a monorepo with all external dependencies vendored.
However, there are at least two cases where you do not want real hermetic builds:
- in the case of open source software that needs to be packaged for distributions, vendoring all the dependencies is bad. As the name already says, distributions remix software components and thus might want to compile your software against a different version of a library than the particular version you pick. - for libraries, consumers use your library just as a component and might want to combine it with a specific version of other libraries, so vendoring also sucks.
Unfortunately, many declarative build systems make it hard to separate the "building a component" part from the "composing a product" part. I could find no way in Bazel to depend on system software (using for example pkgconfig).
In my opinion, build systems for open source software (as I mentioned, for companies Bazel makes sense since you often want to vendor all your deps anyway) need to have three aspects:
- build rules for just for this component - dependencies on other components/libraries - a method to lock down components/libraries to a specific version, creating an offical "product" with an official supported combination of all dependencies.
This still makes it possible for distributions to create their own "product", without having to rewrite most parts of the build system. At the same time, developers can work with the official library versions (for compat, maybe you should even have multiple "sets" of dependencies), so getting started is easy and you don't need to install lots of dependencies first before you can hack on the project.
Often, build systems do things that are nice for onboarding (like "build systems" that pull git repos of dependencies during build, in order to have a single command to build the product from scratch after checkout). But those things directly conflict with the needs of distributions. We need to separate the "instructions to build, supplying all deps externally explictly" (in the context of building a package for a distribution) from the "just build it from the command line on a dev machine" (here, dependencies should be implictly fetched automatically, to give a nice user experience).
You could probably write a rule similar to go_repository[1] that would invoke pkgconfig to find the location of local libs instead of downloading a lib.
[1] https://github.com/bazelbuild/bazel-gazelle/blob/master/inte...
The problem with fully declarative systems is that, if you need something that is beyond the scope of the declarative semantics, you are screwed.
Build systems have the annoying property that they are ultimately highly declarative, but they need some pretty extreme scope to handle edge cases. Any C/C++ project is going to boil down to "compile these N source files, and link them into a library/executable", but the necessary build flags is going to come from several locations, there are multiple variations of build flags and compilers that need to be used at the same time, and there is going to be project-custom code to generate source files that involves potentially building the tools to generate the files as part of the build system. And mixed-language projects add more layers of pain.
bazel in particular is pretty nice with that. There are multiple extension points -- there is embedded Python-like language, you can wrap arbitrary commands, and if this is not enough, you can dynamically general build files as well.
> The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X…but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit to make itself. Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work. After several rounds of these kind of fixes, you wind up with Makefiles that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel. And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be…challenging.
We had a particularly "fun" instance of this sort of Makefile weirdness back when we used Make for Rust... https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d...
(Discussed on HN: https://news.ycombinator.com/item?id=13655081)
Before it comes up: Bazel ships with its own copy of the JVM so you do not need to install Java to use it.
Is this supposed to be a good thing?
Is it a bad thing?
That might make the installed size large, but if it's isolated to Bazel, managed by Bazel, updated by Bazel, etc, then why does it matter whether it's a JVM, a Python binary, a JS VM, or anything else?
Probably better than Nodejs Gulp Yo man ugabuga npm here's 3.4 million files for you to download or Python is it 2.7 or 3.1? Pip too and Ansible let's make that terminal a disco night of text! or some Go "Works great for my static file blog" build tool.
At some point someone will just bundle all the software into a single docker container, and then we are finally done with the entire installation quagmire.
...if you're happy running random third party binaries instead of building from source.
Do you review all of the source-code that you run? If you trust the source-code due to its origin, why not trust a binary?
Firefox will currently build on more platforms than Bazel does. Getting fed up of the endless list of new build systems.
What platforms are you talking about specifically? Bazel builds for Windows, Linux, MacOS, iOS, Android.
Just guessing: BSDs, ARM (eg Raspberry Pi), PPC, Haiku, Minix
I agree, and inflicting Bazel's horrible Java dependency on Firefox would be a tragedy.
What's wrong with the Java dependency? There's even a single-binary Bazel executable[1] that brings its own embedded OpenJDK. My company is using Bazel and the Java dependency has never been an issue.
Even if you write custom rules, you use the Python-like Skylark language and never interact with Java code.
[1]: https://github.com/bazelbuild/bazel/releases (the single-binary distribution is bazel-1.1.0-linux-x86_64, it's only 40MB).
How is it different than depending on e.g. Python?
If making Bazel work on Windows is a side effect of Firefox using it, what's not to like about it.
Of all the build systems I've used, Bazel/Blaze has sucked the least.
Bazel is excellent on Linux and macOS, but it's pretty bad on Windows. I'm not sure what (if anything) they're going to do about that. Maybe they'll fix it on Windows though. That'd be pretty cool.
Microsoft internally uses something pretty similar to Bazel. I'm not familiar enough with the two to fully understand motivations for the divergence. It supposedly initially ran poorly on Mac, just due to differences in what is cheap on different platforms. I wonder what kind of inherent performance differences you would find in something "Windows first" vs "Linux/MacOS first"
https://github.com/microsoft/BuildXL
Never tried it on Windows. For me it was pretty bad on Linux and macOS.
I wonder if they considered build2[1]? It seems to tick most of their boxes. In particular, it has first-class Windows support and, in the upcoming release, even native Clang-targeting-MSVC support (not the clang-cl wrapper)[2]. While some of the features they are looking for (like faster build times which I assume means distributed compilation and caching) are not there yet, we are working on them (and would welcome help)[3].
I think part of the reason is battle-testing. Blaze (the Google internal version of Bazel) has been used by 20k people per day for well over five years to continually build a single mammoth shared C++, Java, Python, Shell (plus others) code base complete with a globally distributed cache system and thousands if not tens of thousands of build machines.
One of the important parts in the post seems to be that it is not just a C++ build system, there are many other tasks executed through the build system. In that sense, it seems like build2 isn't a great fit.
I'm surprised to see no mention of Tup [1] in the discussion. Verifying the dependency graph for maximizing parallelism is its major feature.
It seems dead? Latest commit was over a year ago, and there are over 100 open issues seemingly with no activity.
Firefox currently can be built with tup, although it's still experimental and not used in production.
Does Tup do more than C? Firefox needs to build some Rust things too.
Windows support seems to be very lacking.
What is different about Tup? It's home page contains a lot of text but zero information.
The problem of Bazel is the JVM requirement. Why not to use something native and more lightweight?
You can compile Java to native code.
And in the case you don't want to, it is hardly any different from shipping a bunch of .so/.dll with the application.
I'm trying to wrap my head around whether Bazel is worth looking into for the kind of work loads I typically work in. There doesn't seem like that much benefit if you have single language code base and especially if you already have a single build.
I suppose I see the benefit of this setup if you keep many libraries and want to pin to the latest instead of, an explicit lib version. If you're in a multi-repo system you're probably not doing this but I can see how Bazel could actually make this possible.
Can someone shed some light on the CI story? Can Bazel do code deployment as well? Is that a smart thing to do with Bazel?
Bazel is worth looking into as soon as you have dependency graphs that your language's native build system can't deal with efficiently, or you have multiple languages that have dependencies on each other's artifacts.
A pure Go codebase? Not worth it.
Personally, I use Bazel even for small projects as soon as they involve things like generated code or gRPC.
Yes, there are multiple rulesets for deployments, like rules_k8s[1] and rules_docker[2]. Of course, you can easily build your own custom deployment pipeline.
Is meson not good?
Meson does not have the same feature set as Bazel. Most notably sandboxing and remote execution.
TIL Thunderbird is still a thing.
It’s still one of the best IMAP clients IMO.