Item — HN Reader

Hi! Author of the post here.

One of the most important invariants that we have to maintain is that there is only 1 container running per repl at any given time. We could determine if the machine is shutting down and not proxy the connection, but we wouldn't have a place to proxy it to since we can't be sure that the existing container has finished shutting down. So either way we end up returning an error and the client has to wait until the old container has been destroyed so a new one can be spawned.

3 comments

Disclosure: I work on Google Cloud (and Preemptible VMs).

You’re kind of racing against the clock though. You could instead have a “load balancing” style layer in front, so that while there is no usable session, at least the person’s connection is just “hanging”.

Feel free to send me some email, as we’re looking to make this experience better (both generally and in GKE, specifically).

> One of the most important invariants that we have to maintain is that there is only 1 container running per repl at any given time.

Perhaps when a host is preempted, instead of killing containers at all, you could just add an iptables rule to black-hole all network traffic for the host. Then they are as good as dead (and the host will be forcibly killed soon anyways).

You still want to send SIGTERM or equivalent to the processes so that they can flush their state if needed. Just abandoning ship is often less good than using a few seconds to cleanly shut down (if only to exit and send a clear message that you have).

Right, but in the article they said they don’t care about graceful shutdown.

I forgot they said that (I read this when it was posted a couple days ago), thanks for noting that.

But I still would encourage even a 1s “clean” shutdown. You don’t need to wait for any of this fancy cleanup, but it’s really nice to finish your writes.

Fun story: for Preemptible VMs we started (in Alpha / EAP) with no soft shutdown just to see whether people could handle it (so just immediate power off). Turns out, that if your box is running apt-get upgrade at the time or anything like that, you easily corrupt your boot disk. So, we struggled between “a few seconds” (5, 15) and the “about 30, which is how long a GCE instance takes to boot”. That’s how we ended up with 30: we wouldn’t more than double regular instance creation times at the tail. Nowadays we boot to ssh in 15 seconds!

If you don’t reuse your state, none of this matters. But I’d guess that even just getting to the point of RST’ing the the connections is valuable (so that the clients know to take action, rather than wait a while).

We had shutdown hooks on our preemptive VMs, but we often had cases (at least weekly), where it looked like they failed to run (failing to unregister from cluster). Any explanation?

Do you mean on GKE or directly on GCE? It sounds like you mean GKE (“failed to unregister from cluster”).

We’re looking to fix up the GKE graceful node shutdown, because it’s currently “racy” and doesn’t actually respect the grace period properly (system pods / processes can be shutdown before waiting for user pods, causing you to lose logging or say the kubelet).

Yes, GKE containers, sorry about confusion... sometimes it looks like a node has disappeared without much shutdown work.

Your solution should implement a retry on failure rule, before returning an error to the client. This is standard in proxies like envoy and others.

We do have an internal retry mechanism for certain kinds of errors, but after too many tries it will return an error to the client. Our clients already have to include a robust retry mechanism because we have to deal any sort of network instability.