Overview
Perform checkouts against your Azure Kubernetes cluster before and after implementing changes such as upgrades by using this Bash shell script I wrote.
This script provides output for the following checks:
✅ Sets the kubeconfig context based on the provided cluster name (names must match)
✅ Display addresses of the master and services
✅ Checks running version of control master and available upgrade path
✅ Checks the kubernetes version running on both the control plane and the node pools
✅ Lists all node pools in the cluster
✅ Lists all nodes running across both pools in the cluster
✅ Output count for total # pods running
✅ Gets all pods in the cluster, loops through each and runs a verbose describe outputting values from Node to Status
Link to the script: https://github.com/jksprattler/kubernetes/blob/main/aks/scripts/aks-checkouts.sh
Usage & Sample Output
The script output can be useful during instances where you need to submit a vendor ticket for troubleshooting. I'll also be including the output in my internal ticket tracking the upgrade to validate it was completed.
Here's some sanitized sample output with redactions for brevity from running the script:
❯ ./aks-checkouts.sh my-rg my-cluster
Switched to context "my-cluster".
########################################################
Kubernetes control plane is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443
CoreDNS is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
########################################################
Name ResourceGroup MasterVersion Upgrades
------- --------------- --------------- -----------------------
default my-rg 1.24.9 1.24.10, 1.25.5, 1.25.6
########################################################
Name Location ResourceGroup KubernetesVersion CurrentKubernetesVersion ProvisioningState Fqdn
-------------------- ---------- --------------- ------------------- -------------------------- ------------------- ---------------------------------------------------
my-cluster westus2 my-rg 1.24.9 1.24.9 Succeeded my-cluster-00000000.hcp.westus2.azmk8s.io
########################################################
Name OsType KubernetesVersion VmSize Count MaxPods ProvisioningState Mode
------ -------- ------------------- ---------------- ------- --------- ------------------- ------
user Linux 1.24.9 Standard_DS13_v2 11 30 Succeeded User
system Linux 1.24.9 Standard_DS2_v2 3 30 Succeeded System
########################################################
NAME STATUS ROLES AGE VERSION
aks-user-99999999-vmss000000 Ready agent 20h v1.24.9
aks-user-99999999-vmss000005 Ready agent 20h v1.24.9
aks-user-99999999-vmss000006 Ready agent 20h v1.24.9
aks-user-99999999-vmss000007 Ready agent 20h v1.24.9
aks-user-99999999-vmss00000a Ready agent 20h v1.24.9
aks-user-99999999-vmss00000d Ready agent 20h v1.24.9
aks-user-99999999-vmss00000k Ready agent 20h v1.24.9
aks-user-99999999-vmss00000o Ready agent 20h v1.24.9
aks-user-99999999-vmss00000p Ready agent 20h v1.24.9
aks-user-99999999-vmss00000q Ready agent 20h v1.24.9
aks-user-99999999-vmss00000s Ready agent 20h v1.24.9
aks-system-99999999-vmss000000 Ready agent 21h v1.24.9
aks-system-99999999-vmss000001 Ready agent 21h v1.24.9
aks-system-99999999-vmss000002 Ready agent 21h v1.24.9
########################################################
NODE POD NAMESPACE STATUS AGE
aks-user-99999999-vmss000000 ama-logs-abcde kube-system Running 2023-05-18T15:19:14Z
aks-user-99999999-vmss000005 ama-logs-fghij kube-system Running 2023-05-18T15:46:55Z
aks-system-99999999-vmss000000 ama-logs-klmno kube-system Running 2023-05-18T14:42:46Z
aks-system-99999999-vmss000001 ama-logs-pqrst kube-system Running 2023-05-18T14:46:15Z
#### Redacted for brevity ####
########################################################
Total number of pods running: 217
########################################################
Checking pod: botkube/botkube-99999999-abcde
Node: aks-user-99999999-vmss000000/1.1.1.1
Start Time: Thu, 18 May 2023 10:24:44 -0500
Labels: app=botkube
component=controller
pod-template-hash=99999999
Annotations: <none>
Status: Running
########################################################
Checking pod: default/filebeat-fghij
Node: aks-user-99999999-vmss000007/1.1.1.2
Start Time: Thu, 18 May 2023 10:34:15 -0500
Labels: app=filebeat
controller-revision-hash=99999999
pod-template-generation=3
Annotations: kubectl.kubernetes.io/restartedAt: 2023-04-13T09:36:26-10:00
Status: Running
########################################################
#### Redacted for brevity ####
Hope you found this useful! What kinds of tools do you use in your Kubernetes environment for running checkouts against changes like upgrades?