back
179 comments
The codebase of an internal system our team inherited is pretty old (14 years) and large, and there's a feeling a lot of the stuff is unused (dead code or unused ancient features) and can be deleted. Fortunately, it's a monolith and all that matters is HTTP endpoints defined in a few config files. So we've made a script which downloads all access logs from the live server for the last 3 months and tries to find endpoints with zero or very low usage (it did find quite a few unused or rarely used endpoints). I'm planning the second step to be a script which builds a dependency graph and finds everything which is only reachable from unused endpoints. I wanted it to be a one-time endeavor but this blog post made me think it should be run on a regular basis.
Be very careful using this methodology. This is how you end up deleting stuff only called in rare but critical cases:

- during outage

- at end of year

- during audits

- when the one special customer that paid a fortune for an obscure feature decides to use it

It might be the case that none of this applies to your application, but it’s why “check what got hit recently” (3 months is an eye blink in the business and govt world) is dangerous.

I wonder if APM performance traces could be used to determine unused code paths.

A lot of these tools sample stack traces at intervals, collecting statistics of how often functions are used. If you had enough samples, then with very high confidence you could find code paths that are never used in production.

Happy to see someone applying some science to our field instead of just going with a hunch.

That said, as kortilla already mentioned, in most cases 3 months is too short time to get a complete answer.

IIRC "Principles of Network and System Administration" by Mark Burgess has more about it, but intuitively you have to look for the largest cycles that exist, but there is also, again as mentioned by kortilla, events that doesn't happen on a schedule.

One thing I try to do is, when I come across code that I have to research is to write down in the documentation for the function (Javadoc or similar) why it exist and the conditions when it can be deleted.

You can also try to add some instrumentation code that notifies you when something is called. That way you can end up like me who, every new years day get a weird sms message at 14:00 or 14:01 from a system that no one can find, but at least I know that some system I cannot remember anymore still runs somewhere :-)

Seems a good methodology, a caveat would be the deletion should start in coeff * 14 years from time of watch start, where coeff is a project complexity coefficient. If the watcher doesn't see a part of code called in another coeff * 14 years, then it's probably safe to delete it.
There is a meta-meta situation surrounding really good software management.

You can knock up some code that say solves a specific business problem right now. (meta:0)

But you need an environment that can take a new piece of code and deploy it and test it (meta:1)

how is that code running - this is shadingnfrom production monitoringninto QA and performance (meta:2)

Compare all the running code and its performance against the benefits of replacing code or going back to level 0 and just fixing a business problem (meta:3)

Then this death eater - meta 4 I think.

And to me this is why comments like "software needs to solve business problems" is naive - once you start using software you need more software to manage the software - it's going to grow till it consumes the business.

> once you start using software you need more software to manage the software - it's going to grow till it consumes the business.

You can replace "software" with "people" and you'll end up with a sentence that's equally true.

True. Which is why it's important to not measure the ROI of a feature by the time and resources you spend on it vs. how it increases your revenue.

You also have to include that every further feature is now _more_ expensive than before.

FYI: Sensenmann is the german word for Grim Reaper
And the literal meaning is “Scythe Man” (which, I'm sure you can guess at this point, is a compound noun of “Sense” for scythe, and “Mann” for man. Both the German Sense and English Scythe derive from the proto-germanic Sagu meaning to cut or saw.)
I still don't get the tech industries fascination with random german words at least here it's sort of fitting.
They should also look at deleting third_party directories and replacing them with package manager dependencies. For example Qt WebEngine embeds Chromium, which embeds Blink, which embeds wpt, which embeds a bunch of Python libraries and one of those Python libraries (six) is embedded in three other locations within Chromium.

https://sources.debian.org/src/qtwebengine-opensource-src/5.... https://codesearch.debian.net/search?q=package%3Aqtwebengine....

I wonder how many copies of the Python six module the Google monorepo contains.

I’d assume one. The “one version policy” is one of the strictest codebase rules at Google.

But the Chrome codebase is generally the giant exception to everything due to the major open source components (Android too).

And while six is 'kinda' well behaved, trying to unify dependencies of dependencies is an exercise in frustration usually. Code repetition is cheap
This is a good plan but I've often found that with for example, Debian, the official packages are well behind the times compared to the direct third party code.

Ubuntu is not too bad.

The most important part of this is that the build units are hermetic and all dependencies are explicit. This is why you need to use something like Bazel/Blaze vs older build systems like make where identifying what's used, particularly when you get into meta-rules, becomes all but impossible.

As the article points out, you also have to look at what's actually run. This is the real advantage of Google infrastructure: the vertical integration so if a binary is run on Borg, or even on the command line, that can be tracked.

I worry about archival and enough history for diagnosing long-standing subtle issues that take a long time to surface as bugs. This is not theoretical: apparently a TeX bug picked up after many years had been there from the start.
i think piper saves the full history of the whole monorepo; if that's correct it's not 'deletion' in that sense
Sounds like a useful system from which almost nothing is usable outside of Google.
Sure; but working at another very large company, I can say I wish we had this.

Old unused code is a huge problem for us. The coordination costs of trying to update company wide problems are made much more severe by old code.

I wish we had something like this. We’re large enough we’d need our own system anyway. We don’t have a monorepo, and we don’t use tools so many others do.

Sure, but this kinda tool only works because of their monorepo, build system, deployment run tracking etc. Let's say they open sourced it, you'd have to basically adopt the entire google stack to make use of it
Code deletion is much easier when you have a typed language with no meta programming. I guess Go and C code is easier to tree shake like that, than say Python or JS.
Sensenmann is able to delete whole libraries independent of programming language due to how the dependencies are defined.

Programming languages would make more of a difference if deletes were happening at a more granular level, e.g. deleting unused functions, but this article doesn't touch on that.

C is certainly not very strongly typed.
They may use Blaze (Bazel) for every language to allow for tracking all dependencies. This helps with reducing the costs of CI by running tests specific to the changed code, and aids code deletion.
It's not that simple. A lot of dead code doesn't look dead to static analysis. E.g. gated by long-forgotten feature flags, or platform checks that are no longer relevant.
This is only the simplest possible reaper tool. You can (and Google does) also write tools to find flags that have only ever been set to one value, codepaths that haven't been exercised in months, databases that haven't been accessed in years, etc.
>For example, if an engineer is unsure how to use a library, they can find examples just by searching

Isn't that the case with all libraries? How does the monorepo help here?

Discoverability. It's a lot easier to search one repo then it is to search a set of repos. For the latter you need to have all the repos listed somewhere, and have them be accessible.
"searching" how a method is used is as simple as clicking on the symbol.

Think Visual Studio "find all references", but working around the entire company's codebase, not just your current project.

I’m not sure. I also thought that these big repos have to use sparse checkouts to not use too much space on the developers machines. So you would have to use an external code search index anyway.
On a non-google level just being aware of code sitting around costing resources is pretty important. Often, tests and maintenance are just ignored or not calculated in as cost (be it time, money, effort or otherwise). It is almost in the same realm as "I don't know why it works" which is as dangerous as "I don't know why it doesn't work".
The most difficult part about code deletion is practicing the Chesterton's Fence principle:

> In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.

https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence

While this tool certainly does the job of proposing code deletions, that's the easier part. The harder part is knowing why the code exists in the first place, which is necessary to know whether it's truly a good idea to remove it. Google, smartly, is leaving that part up to a human (for now).

Conducting an inventory (of services) would be cool too.

If a chunk of code isn't actually deployed somewhere, mark it as a candidate for culling.

Probably requires some kind of metadata provenance for deployed artifacts.

For Java, I thought Maven had a stock manifest.mf entry for the source repo. Alas, a quick search only reveals that the archiver plugin has an entry for the project's "url".

https://maven.apache.org/shared-archives/maven-archiver-2.5/...

Which for in-house projects is probably sufficient.

I'm generally not fond of monorepos but reading this and right away being targeted by this tweet https://twitter.com/nikitabier/status/1652764613962760196 made me think on how much time goes on decisions that won't impact users directly and how hard it becomes to justify that on regular companies in an environment where CEOs are jumping the layoff bandwagon for no reason.
"deleting code" should have been included in the "two hardest things in computer science".
Sincere quesion: what is interesting or novel about this? Is it just the scale or did I miss some subtle aspect?

This is more (or less?) the same as industry best practices, just scaled up. There is a challenge in scaling up, as there is more potential for someone to mess it up. But it's the same technique.

So what am I missing?

Vert interesting. At our company we have a spring cleaning event next week, and one of the goals is to delete old code. Some of the comments mention some good techniques. Anyone experienced with deep analysis, e.g. BPF on this field?
It's about time someone gets a KPI for lines of code removed :)
I liked most of the blog, but it bothers me to see stuff like, "... just as with the introduction of unit testing 20 years ago...".

No, unit testing was NOT introduced 20 years ago. As an example, Perl 1 was released about 35 years ago with a unit test suite that got run on every install. Every version of Perl has done so since, and since CPAN came along, most Perl modules have followed suit. This was the secret sauce behind Perl's reputation for being so portable.

Nor was Perl a pioneer. In fact unit testing was used in the 1960s on the Apollo program, and was even called unit testing. I believe that the concept can be dated back to a 1950s textbook but I can't find the reference.

So unit testing is over 60 years old.

I am guessing it would be very hard to do this on a dynamic language.
Looks like graph algorithms comes in handy at the end
The Devel's in the Details: typo or pun?
Does this only get rid of unused binaries? Or is the system smart enough to use the profiling infrastructure to identify dead code in general?
I hope everyone involved gets L+1!
Upvoted for the nice name :)