Troubleshooting Clusters [page]deterministic

This doc is about cluster troubleshooting; we assume you have already ruled out your application as the root cause of the problem you are experiencing. See the [application troubleshooting guide](/docs/tasks/debug/debug-application/) for tips on application debugging. You may also visit the [troubleshooting overview document](/docs/tasks/debug/) for more information.

For troubleshooting [kubectl](#gloss:kubectl), refer to [Troubleshooting kubectl](/docs/tasks/debug/debug-cluster/troubleshoot-kubectl/).

## Listing your cluster

The first thing to debug in your cluster is if your nodes are all registered correctly.

Run the following command:

```shell kubectl get nodes ```

And verify that all of the nodes you expect to see are present and that they are all in the `Ready` state.

To get detailed information about the overall health of your cluster, you can run:

```shell kubectl cluster-info dump ```

### Example: debugging a down/unreachable node

Sometimes when debugging it can be useful to look at the status of a node -- for example, because you've noticed strange behavior of a Pod that's running on the node, or to find out why a Pod won't schedule onto the node. As with Pods, you can use `kubectl describe node` and `kubectl get node -o yaml` to retrieve detailed information about nodes. For example, here's what you'll see if a node is down (disconnected from the network, or kubelet dies and won't restart, etc.). Notice the events that show the node is NotReady, and also notice that the pods are no longer running (they are evicted after five minutes of NotReady status).

```shell kubectl get nodes ```

```none NAME STATUS ROLES AGE VERSION kube-worker-1 NotReady <none> 1h v1.23.3 kubernetes-node-bols Ready <none> 1h v1.23.3 kubernetes-node-st6x Ready <none> 1h v1.23.3 kubernetes-node-unaj Ready <none> 1h v1.23.3 ```

```shell kubectl describe node kube-worker-1 ```

```none Name: kube-worker-1 Roles: <none> Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=kube-worker-1 kubernetes.io/os=linux node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 17 Feb 2022 16:46:30 -0500 Taints: node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unreachable:NoSchedule Unschedulable: false Lease: HolderIdentity: kube-worker-1 AcquireTime: <unset> RenewTime: Thu, 17 Feb 2022 17:13:09 -0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 17 Feb 2022 17:09:13 -0500 Thu, 17 Feb 2022 17:09:13 -0500 WeaveIsUp Weave pod has set this MemoryPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Addresses: InternalIP: 192.168.0.113 Hostname: kube-worker-1 Capacity: cpu: 2 ephemeral-storage: 15372232Ki hugepages-2Mi: 0 memory: …(trimmed)