Item — HN Reader

> The confidence intervals are represented as latency ± (lower, upper), in which latency corresponds to the fifth percentile of latency measurements across 100 different machines. Lower and upper refer to the bounds of the 95% confidence interval for this percentile.

Does anybody know why they chose fifth percentile? I though we should always choose the fastest time when measuring performance.

2 comments

Usually you discard extreme values to reduce noise, and in fact they wrote that's why they did it:

> We then take the fifth percentile as our final measurement, because we assume that most noise sources are one-sided (for example, cache misses, pre-emptions and so on). During training we process the measurements across ten machines for computational efficiency.

> I though we should always choose the fastest time when measuring performance.

Depends. For games you usually do sth similar to what they did - exclude small percentage of worst results to reduce influence of noise and then optimize the worst scenario to make the game run consistent and smooth.

One possibility which seems not so well-known is that clocks with per-core state might not be perfectly synchronized. If your initial measurement is from core0, then we migrate to core1, the end measurement could even be 'before' the initial.

Then there are manufacturing differences between cores that affect e.g. their leakage current and thus the (turbo) frequency at which they can run.

So the measurement noise is indeed not one-sided, that is to say: measurements are not always overestimates. Thus a trimmed mean on both sides is a good idea, and pinning threads to a core when measuring is also helpful.

Because they want to make sure that the sorting algorithm works well for all possible workloads, not just the most preferable ones.

If we measured sorting algorithms by the fastest measurement, we might conclude that BubbleSort is the fastest possible sort algorithm on some inputs. (Bubblesorting an already-sorted list makes at most one comparison per list element)

I don't think that's what they meant (or I have misunderstood). Running the same algorithm on the same input still has variations because of OS/CPU idiosyncrasies. When measuring performance we usually run the algorithm on the same input multiple times and report the fastest performance.