A story about deprecated Kubernetes labels and software startup failures
I would like to share a true story which happened recently at my work which led to the massive frustration and was caused by the mixture of accumulated technical debt over few years, lack of proper validation in place and few other things.
Let me start first with the situation we ran into. We have a lot of teams distributed around the world, implementing sub-components of the integrated solutions, all based on RedHat OpenShift Container Platform. We support all the existing architectures (x86, Power, s390x), however some sub-components might support just subset of those (e.g. we have components which could be deployed only on x86).
As we wanted to be a good citizen in the k8s cluster, we are using nodeSelectors and node affinity rules, to specify that we want our apps (Pods) to be scheduled only on the worker nodes which match the desired architecture. Readers might argue whether it’s the best approach to do the job, yet here’s a snippet how we do this:
Readers might say it’s good or bad thing, yet it works good for most of the cases, for several years. However, recently we found out that on OpenShift v4.6, after the cluster restart (like power-off/power-on scenario), application is not properly starting, showing Pods stuck in the Predicate NodeAffinity failed status and the only way to solve the issue is to manually kill and start those Pods. Not ideal solution for the enterprise-grade software and BAU operation like cluster restart, huh?
It took our team quite some time the root cause of this problem and we found the issue reported in the k8s community, which describes the actual situation happening. Geeky readers can follow the linked details, yet the high level description is that on cluster restart, the beta.* labels are added into the worker nodes with a delay, while Pods are marked for execution already, yet the affinity rule is not met and Pods are stuck (simplified view of the situation). The issue is intermittent. To add a little bit more spice to this story, the initial template for the nodeAffinityPod definitions was proposed by our team several years ago and worked like a charm for quite some time.
What is turned out was that actually, the labels we were using are deprecated since k8s v1.14, were supposed to be removed in k8s v1.19, yet we were living on the borrowed time. The fix was easier than you though: when the definition was updated to use kubernetes.io/arch , application starts just fine after cluster restart.
Lessons learnt?
Whenever you use any alpha or beta APIs (yes, labels are also APIs), make sure you have some process in place to review carefully k8s release notes, to see whether your APIs becomes deprecated. Otherwise, you might hit the wall sooner or later — in our case we ran into intermittent issue, but if we waited for k8s v1.22, our app would not start at all, so — BE WARNED!
