Mostly you just have to do what is outlined in the next few paragraphs. Optimization is actually pretty simple - don't work against the hardware, work with it. Measure.
Memory is slow. Just changing how things are ordered in the memory can do sometimes 2x speed. So try to keep your data sequential in memory. Avoid locks if you can - it's shared memory between cores which requires a lot of slow communication between cores. Avoid atomic operations if you can - also memory shared between cores. Avoid sharing any mutable data with other cores, if you can. If NUMA (multiple physical CPU sockets, each with it's own physically connected memory), use local CPU memory, otherwise you'll very quickly saturate CPU socket interconnects! Avoid function pointers, especially to short functions; they cause compilers to drop optimizations and cause often CPU branch mispredicts. Avoid pointer chasing, but if you do, measure if it makes sense to early prefetch next entry/entries. Don't waste cache lines! You have less than 512 or 1024 of those very valuable L1 cache lines, they're easy to exhaust. Don't write unnecessarily to cache lines shared by other cores, this makes a lot of very slow things happen behind the scenes. Try to align structs to next larger power of two size, but do remember often just having more data in cache is better, so measure. Or align structs to cache line boundary (usually 32, 64 or 128 bytes), so that CPU doesn't need to fetch two cache lines. Otherwise align to data type size, but measure too. Try to keep your stack aligned.
Don't register spill, be frugal with how many registers you need in inner loops (especially on 32-bit x86!). Don't do long dependency chains in computations, if you can avoid it.
Minimize the number of branches, even nearly 100% predictable ones - they still take up valuable CPU branch resources.
If you have a lot of memory random access, arrange your data in a way that maximizes locality of reference. Use large pages to avoid TLB misses (and worse, page faults).
Noticed a pattern? Computation is fast, memory is very slow. Sequential streaming from memory is faster by an order of magnitude compared to random access. You might be able to do thousands of floating point operations during a single cache miss! Communicating with other cores is slower. A lot of effective optimization comes down to managing cache better.
Always assume what you know is outdated and even when it isn't, that you're simply wrong! Read CPU model specific documentation, but don't trust it blindly. Assume nothing, measure. Small changes in code can affect performance in seemingly unaffected locations. Don't do silly things: it doesn't matter if your loop counters count up or down. "register" keyword does nothing and even if it did, you'd probably make your code slower.
Be sure to measure on multiple CPU architectures (for X86: a few AMD generations, Intel Sandy Bridge, Atom, etc. All of those behave differently, often very differently! Take advantage of the CPU performance counters, they're really useful seeing what happens inside the chip.
Don't optimize until you're really sure you need to, and that you've done all you can in choosing the best algorithm. Is everything really necessary code to execute? Anything you could precompute? Do remember that you can do a huge amount of computation during a memory access! Don't naively assume you should always choose the algorithm with the best big O. Your constants are often large and data is usually small!
> ARM isn't really simpler than x86. ARMv7 has about as many instructions as x86, and probably has more if you differentiate between thumb and arm (which you should if you are micro-optimizing). It also features some constructs like general conditional execution that are notoriously difficult for compilers to use effectively. The 64-bit ARM ISA is every bit as complex as modern x86_64 (though more orthogonal, which makes it lovely to program for).
True. ARMv7 conditional execution (predicates) is also reading from flags register. On a simple pipelined ARM implementation that equals a stall, if flags were touched by the few previous instructions, until last flag touching instruction is retired. And on more complicated designs that implement out-of-order execution it can still often mean a stall because of long dependency chains.
On ARMv7 practically all instructions can have dependency on flags register!
You do avoid branch mispredicts, but at what cost? Having so many read/write ports in the flags register consumes silicon and power too. The cost increases the deeper the pipeline. To get higher clock frequencies, the CPU pipeline needs to become longer.
ARMv8 (64-bit) did the only sensible thing and removed predicate insanity.