Reader/Reader blocking in reader/writer locks

I encountered this issue a year or two ago on the Netflix CDN nodes when running with 100K TCP connections or more. We have metrics collectors that need to report how many TCP connections are open, and what state they are in (ESTABLISHED, TIME_WAIT, etc). Until recently, the only way to do this was to ask the FreeBSD kernel to copy out the entire TCP hash table to userspace, where an application can examine it & count connections. The problem with that is that the thread wanting a copy of the table had to take an rlock on the table while it was being copied. As the author points out, this blocks any writers, which in turn blocks more readers. So we'd see latency spikes when this script was running.

The problem has been fixed twice: by moving the top-level locking for that table to epoch (FreeBSD RCU equivalent), and also by using per-cpu counters to track the number of TCP connections in every state. So now the user-space collector just needs to fetch a small array of tcp states from the kernel.

This is a decent reminder blog post about something that is obvious on the surface, but subtle when you forget to think about it and make a mistake.

I've found that thinking of a lock as something you don't want to hold goes a long way. The point of locks is to release them.

Jeff Preshing has a good series on locks and concurrency. This[0] post (potentially with an extra dozen I read in the same sitting) is the one that really changed how I thought about locks: before I read it, I was very into lock freedom at every opportunity.

0: https://preshing.com/20111118/locks-arent-slow-lock-contenti...

> So reader/writer locks tend to rely on the additional concurrency, and on readers not blocking other readers.

> software has a constant tendency to get slower, absent deliberate and careful efforts to hold the line, as developers add features and complexity. And if slower processes don’t immediately impact performance (because they only hold read locks), they’re more likely to go unnoticed. Thus, it’s not unlikely that over time read lock durations will creep upwards, mostly without effect, until one happens to coincide with an attempt to grab a write lock.

For me, the takeaway here is to maintain discipline in software regarding locking behavior. It's almost always a bad idea to hold a lock (rw or not) for an extended period of time.

Notably, this is how the Linux kernel developers responded to the case linked in the discussion of mmap_sem - the operation quadratic in number of threads was thrown out to avoid holding the rwsem for extended periods of time.

Also, re "and the lock is held by readers for a significant length of time": I would imagine that Rusty Russell would consider a few microseconds a "significant length of time" and wonder if you were insane if your software held a lock for 60 seconds.

> What to do about it?

Design with RCU, if you can.

> Time out writer acquisitions.

> If potentially starving writers is acceptable, adding a timeout to write-side acquisitions will bound how long readers can be forced to pause. GoCardless’ library for online Rails migrations implements this option for Postgres, and also discusses the challenges and implications a bit.

This is nearly the same as simply not having writer priority. If "potentially starving writers is acceptable", that implies we can consider reader-priority locks, which do exactly that: potentially starve writers.

Under writer priority, we hold back new readers so that the old readers can leave and let the waiting writer in. If we balk when the new readers are blocked for too long, and bump the writer, then it's simply not writer priority any more.

Please make the second "reader" in the title lowercase, otherwise it's very confusing to parse (looks like three things separated by slashes, instead of a single sentence with two words that have slashes in them).

For the database example, could you change your isolation level for the reading queries?