We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match.

Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker.

Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.

Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit.

So far our solution has been to just remove CPU limits, but hoping things will get better.

Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary