How to: Find ownerReferences of Kubernetes workloads

5 min readFeb 7, 2022

This is another article on my kubernetes learning series. I keep posting my interesting learnings that may be useful someone who’s trying to master kubernetes.

Background

I work on large clusters-of-clusters which contain multiple individual kubernetes clusters deployed by a single kubernetes cluster. Most often, one or the other server hits a hardware fault and I have to offload workloads from that server to shutdown and perform hardware maintenance.

The procedure of draining-cordoning-shutdown of the node is automated through Ansible playbooks. However, I always saw some pods still running after draining step and wondered why. It wasn’t much tough to figure out that all pods belonged to some daemonset and also the playbook drained the nodes with ``` — ignore-daemonsets``` . Hower this kept changing when I switch to another cluster, where the workloads are different. So I decided to figure out which pods will stay after draining, i.e., find the relationship of a pod to a workload(deployment/replicaset/daemonset/statefulset).

Kubernetes API coming to the rescue

There was no straightforward way. One tiresome way is to find workloads, match the pods with their labels. But is it how kubernetes manages the relationship? There should be a reference of this relationship somewhere in the keystore(etcd) right? So I decided to dump everything in etcd and explore it. But before that I wanted to give a try on the kubectl get verbose output. We can increase verbosity from level 1 to 10.

gosekar@gosekar-k8s-master:~ $ kubectl get pods nginx-8jw4r -v=10
I0207 11:52:06.430292  699507 loader.go:372] Config loaded from file
:  /home/gosekar/.kube/config
[...]
I0207 11:52:06.455951  699507 round_trippers.go:435] curl -v -XGET  
-H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application
/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Ag
ent: kubectl/v1.22.1 (linux/amd64) kubernetes/632ed30" 'https://10.1
27.200.247:6443/api/v1/namespaces/default/pods/nginx-8jw4r'
I0207 11:52:06.458249  699507 request.go:1181] Response Body: {"kind
[...]
org/podIPs":"10.244.234.104/32"},"ownerReferences":[{"apiVersion":"v
1","kind":"ReplicationController","name":"nginx","uid":"4a81543e-625
6-4a26-8310-a8eecef6b438","controller":true,"blockOwnerDeletion":tru
e}],"managedFields":[{"manager":"kube-controller-manager","operation
[...]

Voila! ownerReference array!

Okay, Now, there’s something we need on the API response. Now I can ask kubectl to specifically give me the output in JSON, so that I can extract desired keys.

Bonus: Kubernetes has a command extension to explain the spec/Kubernetes API structure.

gosekar@gosekar-k8s-master:~ $ kubectl explain pods.metadata.ownerReferences KIND:     Pod
VERSION:  v1
RESOURCE: ownerReferences <[]Object>
DESCRIPTION:
     List of objects depended by this object. If ALL objects in the 
     list have been deleted, this object will be garbage collected. 
     If this object is managed by a controller, then an entry in 
     this list will point to this controller, with the controller 
     field set to true. There cannot be more than one managing 
     controller.
     OwnerReference contains enough information to let you identify 
     an owning object. An owning object must be in the same 
     namespace as the dependent, or be cluster-scoped, so there is 
     no namespace field.
FIELDS:
   apiVersion   <string> -required-
     API version of the referent.
   blockOwnerDeletion   <boolean>
     If true, AND if the owner has the "foregroundDeletion" 
     finalizer, then the owner cannot be deleted from the key-value
     store until this reference is removed. Defaults to false. To 
     set this field, a user needs "delete" permission of the owner,
     otherwise 422 (Unprocessable Entity) will be returned.
   controller   <boolean>
     If true, this reference points to the managing controller.
   kind <string> -required-
     Kind of the referent. More info
   name <string> -required-
     Name of the referent. More info
   uid  <string> -required-
     UID of the referent. More info

Filtering what I wanted

jq is handy command line tool for operating on json data streams and is present on most modern UNIX/Linux systems.

kubectl get pods -A -o json | jq -r ‘.items[].metadata | [.name, .namespace, .ownerReferences[].kind, .ownerReferences[].name] | @tsv’ | column -t

So we can print the ownerReference for all pods in all namespaces and can use this as a learning tool for understanding workload relationship/dependencies in a cluster quickly (especially when you have hundreds of pods).

Kubectl — inbuilt Swiss army knife for Kubernetes

Like openstack clients, I tried to see if kubectl is helpful in controlling the output format, beyond giving json/yaml outputs and to my expectation, kubectl did not disappoint me. kubectl has options to extract specific keys from Kubernetes API response and print them with your own column name in the header too! Plus, you can sort it with another key (not more than one key for sorting now)

Needless to say, I tried to run every type of workload kubernetes offers.

More learnings

Every pod relates to one or the other kubernetes workload, but there were few pods with owner as ‘Node’. This is confusing when we have daemonset for the same purpose i.e, running pods in every node. These are the static pods that are run by node’s kubelet instance. These static pods are special in some manner compared to normal pods. For example, these are not monitored by control-plane but by kubelet directly, pods are created jusst by keeping the spec file at a particular folder(which is periodically scanned by kubelet).

gosekar@gosekar-k8s-master:~$ cat /proc/`pgrep kubelet`/cmdline
/usr/bin/kubelet--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kub
elet.conf--kubeconfig=/etc/kubernetes/kubelet.conf--config=/var/lib/
kubelet/config.yaml--network-plugin=cni--pod-infra-container-image=k
8s.gcr.io/pause:3.5
gosekar@gosekar-k8s-master:~$ grep -i static /var/lib/kubelet/config
.yaml
staticPodPath: /etc/kubernetes/manifests
gosekar@gosekar-k8s-master:~$ ls -l /etc/kubernetes/manifests
-rw------- root root Aug 18 18:10 etcd.yaml
-rw------- root root Aug 18 18:10 kube-apiserver.yaml
-rw------- root root Aug 18 18:10 kube-controller-manager.yaml
-rw------- root root Aug 18 18:10 kube-scheduler.yaml

Also this answered my doubt — Which came first: Chicken or egg?. So the kubelet spawns the control-plane pods and upon being up, control-plane pods spawn other pods.
More about statics pods at https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/

The ownerReference object is not just for pods. It’s there for any object that has a dependency. For example, in kubernetes, there can be standalone replicasets and also replicasets that are internally created on creation of a deployment. In the same way, Cronjob created jobs internally and so on. We can use the same command for listing such references if the object has such a provision.

gosekar@gosekar-k8s-master:/tmp$ kubectl get rs -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER-KIND:.metadata.ownerReferences[].kind,OWNER-NAME:.metadata.ownerReferences[].name,NODE:.spec.nodeName,STATUS:.status.phase | (sed -u 1q; sort -k 3)
NAMESPACE              NAME                                    OWNER-KIND   OWNER-NAME
kube-system            calico-kube-controllers-58497c65d5     Deployment   calico-kube-controllers
kube-system            coredns-78fcd69978                     Deployment   coredns
kubernetes-dashboard   dashboard-metrics-scraper-856586f554   Deployment   dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard-67484c44f6        Deployment   kubernetes-dashboard
default                nginx-deployment-66b6c48dd5            Deployment   nginx-deployment

Thanks for reading all the way down here! happy learning.