Understand Pressure Stall Information (PSI) Metrics [page]deterministic

As a stable feature, Kubernetes lets you configure the kubelet to collect Linux kernel [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html) (PSI) for CPU, memory, and I/O usage. The information is collected at node, pod and container level. Starting with Kubernetes v.1.36, the `KubeletPSI` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is locked to true and cannot be disabled.

PSI metrics are exposed through two different sources: - The kubelet's [Summary API](/docs/reference/config-api/kubelet-stats.v1alpha1/), which provides PSI data at the node, pod, and container level. - The `/metrics/cadvisor` endpoint on the kubelet, which exposes PSI metrics in the [Prometheus format](/docs/concepts/cluster-administration/system-metrics#psi-metrics).

### Requirements

Pressure Stall Information requires the following on your Linux nodes:

• The Linux kernel must be version 4.20 or newer.
• The kernel must be compiled with the `CONFIG_PSI=y` option. Most modern distributions enable this by default. You can check your kernel's configuration by running `zgrep CONFIG_PSI /proc/config.gz`.
• Some Linux distributions may compile PSI into the kernel but disable it by default. If so, you need to enable it at boot time by adding the `psi=1` parameter to the kernel command line.
• The node must be using [cgroup v2](/docs/concepts/architecture/cgroups).

## Understanding PSI Metrics

Pressure Stall Information (PSI) metrics are provided for three resources: CPU, memory, and I/O. They are categorized into two main types of pressure: `some` and `full`.

* `some`: This value indicates that some tasks (one or more) are stalled on a resource. For example, if some tasks are waiting for I/O, this metric will increase. This can be an early indicator of resource contention. * `full`: This value indicates that *all* non-idle tasks are stalled on a resource simultaneously. This signifies a more severe resource shortage, where the entire system is unable to make progress.

Each pressure type provides four metrics: `avg10`, `avg60`, `avg300`, and `total`. The `avg` values represent the percentage of wall-clock time that tasks were stalled over 10-second, 60-second, and 5-minute moving averages. The `total` value is a cumulative counter in microseconds showing the total time tasks have been stalled.

Let's take for example the following query from the Summary API: `kubectl get --raw "/api/v1/nodes/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')/proxy/stats/summary" | jq '.pods[].containers[] | select(.name=="<CONTAINER_NAME>") | {name, cpu: .cpu.psi, memory: .memory.psi, io: .io.psi}'`. This returns the information in a json format as such.

``` { "name": "<CONTAINER_NAME>", "cpu": { "full": { "total": 0, "avg10": 0, "avg60": 0, "avg300": 0 }, "some": { "total": 35232438, "avg10": 0.74, "avg60": 0.52, "avg300": 0.21, }, }, "memory": { "full": { "total": 539105, "avg10": 0, "avg60": 0, "avg300": 0 }, "some": { "total": 658164, "avg10": 0.01, "avg60": 0.01, "avg300": 0.00, }, } }, "io": { "full": { "total": 33190987, "avg10": 0.31, "avg60": 0.22, "avg300": 0.05, }, "some": { "total": 40809937, "avg10": 0.52, "avg60": 0.45, "avg300": 0.12, } } } ```

Here is a simple spike scenario. The cpu.some `avg10` value of `0.74` indicates that in the last 10 seconds, at least one task in this container was stalled on the CPU for 0.74% of the time (0.0074 seconds or 74 milliseconds). Because `avg10` (0.74) is significantly higher than `avg300` (0.21) on the same resource, this suggests a recent surge in resource contention rather than a sustained long-term bottleneck. If monitored continuously and the `avg300` metrics increase as well, we can diagnose a more serious, lasting i …(trimmed)