Troubleshooting Topology Management [page]deterministic

Kubernetes keeps many aspects of how pods execute on nodes abstracted from the user. This is by design. However, some workloads require stronger guarantees in terms of latency and/or performance in order to operate acceptably. The `kubelet` provides methods to enable more complex workload placement policies while keeping the abstraction free from explicit placement directives.

You can manage topology within nodes. This means helping the kubelet to configure the host operating system so that Pods and containers are placed on the correct side of inner boundaries, such as _NUMA domains_. (NUMA is an abbreviation of _non-uniform memory access_, and refers to an idea that CPUs might be topologically closer to specific regions of memory, due to the physical layout of the hardware components and the way that these are connected).

## Sources of troubleshooting information

You can use the following means to troubleshoot the reason why a pod could not be deployed or became rejected at a node, in the context of topology management:

• _Pod status_ - indicates topology affinity errors
• _system logs_ - include valuable information for debugging; for example, about generated hints
• _kubelet state file_ - the dump of internal state of the Memory Manager (including the _node map_ and _memory maps_)
• You can use the [device plugin resource API](#device-plugin-resource-api) to retrieve information about the memory reserved for containers

## Troubleshoot `TopologyAffinityError` {#TopologyAffinityError}

This error typically occurs in the following situations:

* a node has not enough resources available to satisfy the pod's request * the pod's request is rejected due to particular Topology Manager policy constraints

The error appears in the status of a pod:

```shell kubectl get pods ```

```none NAME READY STATUS RESTARTS AGE guaranteed 0/1 TopologyAffinityError 0 113s ```

Use `kubectl describe pod <id>` or `kubectl events` to obtain a detailed error message:

```none Warning TopologyAffinityError 10m kubelet, dell8 Resources cannot be allocated with Topology locality ```

## Examine system logs

Search system logs with respect to a particular pod.

The set of hints generated by CPU Manager should be present in the logs. Also, the set of hints that Memory Manager generated for the pod can be found in the logs.

Topology Manager merges these hints to calculate a single best hint. The best hint should also be present in the logs.

The best hint indicates where to allocate all the resources. Topology Manager tests this hint against its current policy, and based on the verdict, it either admits the pod to the node or rejects it.

Also, search the logs for occurrences associated with the Memory Manager; for example to find out information about `cgroups` and `cpuset.mems` updates.

## Examples

### Examine the memory manager state on a node

Let us first deploy a sample `Guaranteed` pod whose specification is as follows:

```yaml apiVersion: v1 kind: Pod metadata: name: guaranteed spec: containers: - name: guaranteed image: consumer imagePullPolicy: Never resources: limits: cpu: "2" memory: 150Gi requests: cpu: "2" memory: 150Gi command: ["sleep","infinity"] ```

Next, log into the node where it was deployed and examine the state file in `/var/lib/kubelet/memory_manager_state`:

```json { "policyName":"Static", "machineState":{ "0":{ "numberOfAssignments":1, "memoryMap":{ "hugepages-1Gi":{ "total":0, "systemReserved":0, "allocatable":0, "reserved":0, "free":0 }, "memory":{ "total":134987354112, "systemReserved":3221225472, "allocatable":131766128640, "reserved":131766128640, "free":0 } } …(trimmed)