Item — HN Reader

The problem is that if you don't know what is required to write fast code you won't design your program to be performant from the start. Lots of applications can be sped up dramatically by rearchitecting them, but you're not going to hit on the right architecture without thinking really hard about memory layout, data access, network usage, etc.

It's not really true that 80% of the code doesn't matter. 80% of the code might not matter the way it is currently designed, but that doesn't mean a different design isn't dramatically better. Benchmarking and profiling doesn't help with that. Real expertise is required and that is developed over time by people who care about improving performance and make it a priority.

2 comments

80% does not matter, period. I mean 20% is a lot. Hotspots are - spots. Like 0,5% of your code. Not UI for example.

You know, when money is flowing in, you have to hire some good people, let them spend time on fixing hotspots and in some rare occasions, rearchitecture a part or maybe two. Probably won’t pay off, but if you are growing fast enough, it will pay 100x.

Also you’re decreasing all future development speed with all this unreadable shit everywhere, so beware of optimizing unnecessary stuff.

For seriously high performance systems like webservers or exchanges, you need to get the architecture right from the start. You can't rearchitect a part or two or solve design problems by making skilled hires later on. The 80% part is a distraction (it's also factually untrue that 80% of code doesn't matter. Some applications don't have hotspots, the performance impact is basically smeared all across the code. It really depends on the application. Sqlite found major speedups by implementing hundreds of little changes that were each almost undistinguishable from noise).

High performance doesn't mean unreadable. Microoptimizations can make code less readable, true, but they also give you the least amount of speedup (in the best case you can get several times speedup if you found some hotspot that can be vectorized, but that's pretty rare). You can get huge performance improvements by optimizing networking and system calls and doing that doesn't make much of an impact on readability.

Aside: this is the deeply unobvious optimisation video "Performance Matters" by Emery Berger: https://www.youtube.com/watch?v=r-TLSBdHe1A - they got a 25% speedup in SQLite in 2019!

Start at 9:30 and watch to 10:40 if you want to jump straight into the entré. Dessert discusses SQLite at 40:25 to 41:25.

Summary: A: Causal analysis is needed to discover opportunities for improvements - they developed a technique and tool. B: Modern CPUs have so many hidden causes of performance variation, that you require special tools to actually measure small performance gains. C: They found a surprising difference between -O2 and -O3 that implied overfitting (false reporting of performance improvement).

I presume these techniques are used in large software companies, but I am guessing many (most?) developers in smaller companies know little about the topic. I still find it unobvious.

As someone who has done performance work for most of my career, I strongly agree with what you've written here. If you have a program that isn't well optimized, it might be true that once you start profiling you can find some hot spots and easy wins, but that doesn't last forever. Earlier in my career I was working on Python web apps, now I write high performance C++ systems, but you can apply this equally to both domains.

Take for example a big and slow Python/Ruby/PHP/whatever web app, something that I'm sure a lot of people on HN have experience with. If you profile one of these apps you're probably going to find that it is slow because it's using an ORM that generates hundreds of SQL queries, the ORM creates tons of intermediate objects which put a lot of stress on the memory allocator and GC, and there will be an endless amount of code that is munging data to take it from one representation (e.g. whatever the ORM returned) to some slightly different representation (whatever the caller actually wanted). There might be a few queries that are particularly expensive, but once you've fixed those you're still going to be left with something big and slow with no obvious path forward to make the code faster. Furthermore the root cause of these problems may not be obvious when looking at profiles because things like memory allocation that are internal to the runtime of your interpreter are typically either not exposed by profiling tools, or if they are it's not clear what action can be taken to improve them.

Likewise if you have a C++ program that does a lot of unnecessary copying and memory allocation, the program is going to be slow because there's a lot of time spent everywhere and there's no one thing to fix. Look at a bottom up profile of a C++ program and see how much time is spent in string constructors, memcpy, etc. and unless you've been thoughtful about this stuff from the start what you find is probably going to be alarming. In fact, since a lot of copy constructors are inlined (including for STL types), with many profiling tools it might not even be obvious that copies are happening at all, since they won't show up in call stacks.

Not every program needs to be high performance, but if you start by writing a lot of code that is slightly inefficient everywhere then it's probably going to be impossible to make things fast if you change your mind down the line.

And not only does the ORM itself have big performance impact, your own code will be designed around the assumptions of this particular ORM, e.g if it is easier to fetch a few row objects from different tables using the ORM and then do some comparison computations in your relatively slow interpreted language rather than writing one fast SQL JOIN the former will almost always be the solution that ends up in production.

ORMs invites you to write bad code, instead of thinking ahead of what queries I need for this specific operation, you just start writing ORM based code aimlessly and try to make it work with the tools available in your language of choice.

And the ORM entities will eventually infect every part of your project and it will almost be impossible to remove them after the fact.

> For seriously high performance systems like webservers or exchanges, you need to get the architecture right from the start.

Agreed - I worked at a company that went through a very painful period of not coping with load. They got there in the end by completely rewriting the core system - lots of people claimed the switch from an interpreted language to a compiled language but the reality was that the rewrite was a completely different architecture.

I often see death by a thousand cuts - everything is slow and there is no single obvious hotspot to fix. We should not assume that performance matters only for a small fraction of the codebase. Sometimes it is the case, sometimes not.

And UI is a bad example IMHO. UI with a high response time which is uncomfortable (or even frustrating) to use is probably the result of thinking that UI performance doesn't matter.

I think sometimes this exact point is lost in the context of a collection of programs that represent a system rather than simply in the context of a single program.

Truly bad UI response time is rarely because of a performance hotspot though. It's almost always unintentionally blocking the UI thread on network requests. I might get mildly annoyed using a sluggish UI, but I get perpetually annoyed whenever I'm doing almost anything on my phone and leave my house's wifi range.

It may be not performance hotspot by which we usually mean CPU intensive code but some other performance problem which typically happens when performance is ignored. Networks don't have infinite bandwidth, zero latency and 100% reliability but I see some people create software pretending that all above is true. It works sometimes (until it doesn't) but can work better had they tried to minimize dependency on remote hosts or make network interactions asynchronous.

My opinion is that performance as well as security is very hard to retrofit if it was not considered from the beginning. And if you have necessary knowledge it doesn't take much more effort to create software which is faster (or more efficient) and secure compare to software created by an ignorant developer.

I heard many times that one should write a program first not thinking about performance, then profile and optimize a few hotspots. But in my experience 2nd part (optimization) rarely happens and people who can profile and optimize usually also can write faster code from the beginning.

It is painful to see how people who don't know and don't want to know typical latency [1] and throughput numbers, who don't know about algorithm complexity and may other things design and implement very inefficient software.

[1] https://gist.github.com/jboner/2841832

> Benchmarking and profiling doesn't help with that.

I've learned that benchmarking and profiling is the _only_ way to write performant code.

I've seen in code review a number of examples of a very fancy algorithm being broken out, and asked, "You realize N is bounded to be at most 100 here?". Or, "you realize the thread overhead here for parallel processing is two magnitudes slower than serial data access on one thread?"

Humans are bad at intuitively understanding where the slow parts of code are. I've seen processing be improved to the point of impossible to grok, shaving 10ms of a processing piece that is 50ms long, only to then spend time blocking waiting for network transfers that require 10s.

I'm of the opinion the biggest performance improvements are usually in design and architecture. What if there were a design that avoided the need to do any network IO? In that case the 10s + 50ms would be a 50ms process, rather than a hard to grok "optimized" 10s + 40ms process. Simple code leads to simpler design, which is easier to reason about and spot the places where things like "this entire network round trip can be cut out", or "we are loading this data multiple times throughout this process, we can load it once", or "we are loading this data and then spending a lot of time querying "n+1", instead if we stored the data in this format with some pre-processing we'll avoid the "n+1" query."

To further rant, the emphasis of algorithms in coding interviews, people enjoying algorithms more than cleaning up crufty architecture - that is the root of a lot of bad software rather. In sum, it's almost always the design that is slow, rarely it's the algorithm. The profiling is key as it let's you know where things are actually slow. (Recently a colleague was trying to optimize a tight loop that processed 1.5M rows. To "optimize" memory usage, they converted all variables to static to 'save' memory and avoid GC pauses. This in effect did _nothing_, the compiler instead was going to inline all the variables anyways and the resulting bytecode was not going to have any extra variables in at all. Converting local variables to static actually made the memory usage just slightly worse. So, this 'optimization' did nothing but make the code worse. A quick benchmark would have shown that optimization having no effect (to really optimize memory usage, we updated the design to stream results to a file rather than keep everything in memory for a final dump to file at the very end). Another example, I once helped a team do performance work for a DB that they spent a year tuning. They did not keep track of any performance benchmarks, what changes did what improvement; and after a year had nothing to show except for a DB that would crash after a few minutes. Taking that over, starting everything over from scratch, benchmarking everything, the project was done a month later and was stupid fast.)

Disagree, once you know what you're looking for you can thread the needle pretty easily. I've worked in high-performance areas most of my career and it's pretty wild when I solve a leetcode problem for fun and can routinely get into the 99% percentile on speed and memory usage just from knowing what to avoid.

I think you are the golf ball balancing perfectly on an upside down bowl. Optimal, but an unstable solution. Most engineers don't yet know because they don't have enough experience (and most engineers haven't worked in high performance for most of their career), so they need the benchmarks and profiling.

Plus, benchmarks are good solely to be able to show your manager that yes, spending 3 weeks on that refactor was indeed useful. Engineers shouldn't need to have to do that, but it is often useful none the less.

strongly agree

it's important to make sure your code base isn't doing anything dumb, like O(n^2) iterations, or 1000 sequential RPCs instead of 1 batch RPC, or etc -- this is i guess your point about architecture

but assuming that bar is cleared, performance optimizations should only be accepted when accompanied by benchmarks/profiles that demonstrate their usefulness at the whole system level