⎈ k8s knowledge compiler

Node-pressure Eviction [page]deterministic

concepts

[definition:node-pressure-eviction]</br>

The [kubelet](#gloss:kubelet) monitors resources like memory, disk space, and filesystem inodes on your cluster's nodes. When one or more of these resources reach specific consumption levels, the kubelet can proactively fail one or more pods on the node to reclaim resources and prevent starvation.

During a node-pressure eviction, the kubelet sets the [phase](/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) for the selected pods to `Failed`, and terminates the Pod.

Node-pressure eviction is not the same as [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/).

The kubelet does not respect your configured [PodDisruptionBudget](#gloss:pod-disruption-budget) or the pod's `terminationGracePeriodSeconds`. If you use [soft eviction thresholds](#soft-eviction-thresholds), the kubelet respects your configured `eviction-max-pod-grace-period`. If you use [hard eviction thresholds](#hard-eviction-thresholds), the kubelet uses a `0s` grace period (immediate shutdown) for termination.

## Self healing behavior

The kubelet attempts to [reclaim node-level resources](#reclaim-node-resources) before it terminates end-user pods. For example, it removes unused container images when disk resources are starved.

If the pods are managed by a [workload](#gloss:workload) management object (such as [StatefulSet](#gloss:statefulset) or [Deployment](#gloss:deployment)) that replaces failed pods, the control plane (`kube-controller-manager`) creates new pods in place of the evicted pods.

### Self healing for static pods

If you are running a [static pod](/docs/concepts/workloads/pods/#static-pods) on a node that is under resource pressure, the kubelet may evict that static Pod. The kubelet then tries to create a replacement, because static Pods always represent an intent to run a Pod on that node.

The kubelet takes the _priority_ of the static pod into account when creating a replacement. If the static pod manifest specifies a low priority, and there are higher-priority Pods defined within the cluster's control plane, and the node is under resource pressure, the kubelet may not be able to make room for that static pod. The kubelet continues to attempt to run all static pods even when there is resource pressure on a node.

## Eviction signals and thresholds

The kubelet uses various parameters to make eviction decisions, like the following:

  • Eviction signals
  • Eviction thresholds
  • Monitoring intervals

### Eviction signals {#eviction-signals}

Eviction signals are the current state of a particular resource at a specific point in time. The kubelet uses eviction signals to make eviction decisions by comparing the signals to eviction thresholds, which are the minimum amount of the resource that should be available on the node.

The kubelet uses the following eviction signals:

| Eviction Signal | Description | Linux Only | |--------------------------|---------------------------------------------------------------------------------------|------------| | `memory.available` | `memory.available` := `node.status.capacity[memory]` - `node.stats.memory.workingSet` | | | `nodefs.available` | `nodefs.available` := `node.stats.fs.available` | | | `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree` | • | | `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` | | | `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` | • | | `containerfs.available` | `containerfs.available` := `node.stats.runtime.containerfs.available` | | | `containerfs.inodesFree` | `containerfs.inodesFree` := `node.stats.runtime.containe …(trimmed)

Sources

concepts/scheduling-eviction/node-pressure-eviction.md · docNode-pressure Eviction

Related (25)

references Kubeletkubelet conf=1
references Pod Disruption BudgetPodDisruptionBudget conf=1
references Workloadworkload conf=1
references StatefulSetStatefulSet conf=1
references DeploymentDeployment conf=1
references Pod PriorityPriority conf=1
defines Node-pressure eviction conf=1
part_of Self healing behaviordescribes conf=1
part_of Eviction signals and thresholdsdescribes conf=1
part_of Eviction monitoring intervaldescribes conf=1
part_of Node conditions {#node-conditions}describes conf=1
part_of Node out of memory behaviordescribes conf=1
part_of Known issuesdescribes conf=1
part_of {{% heading "whatsnext" %}}describes conf=1
part_of Self healing for static podsdescribes conf=1
part_of Eviction signals {#eviction-signals}describes conf=1
part_of Eviction Thresholdsdescribes conf=1
part_of Node condition oscillationdescribes conf=1
part_of Pod selection for kubelet evictiondescribes conf=1
part_of Minimum eviction reclaimdescribes conf=1
part_of Schedulable resources and eviction policiesdescribes conf=1

← all Docs