Handling retriable and non-retriable pod failures with Pod failure policy [page]deterministic

taskssecurity

This document shows you how to use the [Pod failure policy](/docs/concepts/workloads/controllers/job#pod-failure-policy), in combination with the default [Pod backoff failure policy](/docs/concepts/workloads/controllers/job#pod-backoff-failure-policy), to improve the control over the handling of container- or Pod-level failure within a [Job](#gloss:job).

The definition of Pod failure policy may help you to: * better utilize the computational resources by avoiding unnecessary Pod retries. * avoid Job failures due to Pod disruptions (such [preemption](#gloss:preemption), [API-initiated eviction](#gloss:api-eviction) or [taint](#gloss:taint)-based eviction).

You should already be familiar with the basic use of [Job](/docs/concepts/workloads/controllers/job/).

## Usage scenarios

Consider the following usage scenarios for Jobs that define a Pod failure policy : - [Avoiding unnecessary Pod retries](#pod-failure-policy-failjob) - [Ignoring Pod disruptions](#pod-failure-policy-ignore) - [Avoiding unnecessary Pod retries based on custom Pod Conditions](#pod-failure-policy-config-issue) - [Avoiding unnecessary Pod retries per index](#backoff-limit-per-index-failindex)

### Using Pod failure policy to avoid unnecessary Pod retries {#pod-failure-policy-failjob}

With the following example, you can learn how to use Pod failure policy to avoid unnecessary Pod restarts when a Pod failure indicates a non-retriable software bug.

1. Examine the following manifest:

1. Apply the manifest:

```sh kubectl create -f https://k8s.io/examples/controllers/job-pod-failure-policy-failjob.yaml ```

1. After around 30 seconds the entire Job should be terminated. Inspect the status of the Job by running:

```sh kubectl get jobs -l job-name=job-pod-failure-policy-failjob -o yaml ```

In the Job status, the following conditions display: - `FailureTarget` condition: has a `reason` field set to `PodFailurePolicy` and a `message` field with more information about the termination, like `Container main for pod default/job-pod-failure-policy-failjob-8ckj8 failed with exit code 42 matching FailJob rule at index 0`. The Job controller adds this condition as soon as the Job is considered a failure. For details, see [Termination of Job Pods](/docs/concepts/workloads/controllers/job/#termination-of-job-pods). - `Failed` condition: same `reason` and `message` as the `FailureTarget` condition. The Job controller adds this condition after all of the Job's Pods are terminated.

For comparison, if the Pod failure policy were disabled, the Job would retry until reaching the `backoffLimit` (6 failures). Because retries use exponential backoff and, with `parallelism: 2`, failures occur in pairs, the delay between attempts increases with each retry. As a result, this example would take at least 9 minutes before the Job fails.

#### Clean up

Delete the Job you created:

```sh kubectl delete jobs/job-pod-failure-policy-failjob ```

The cluster automatically cleans up the Pods.

### Using Pod failure policy to ignore Pod disruptions {#pod-failure-policy-ignore}

With the following example, you can learn how to use Pod failure policy to ignore Pod disruptions from incrementing the Pod retry counter towards the `.spec.backoffLimit` limit.

> Caution: Timing is important for this example, so you may want to read the steps before execution. In order to trigger a Pod disruption it is important to drain the node while the Pod is running on it (within 90s since the Pod is scheduled).

1. Examine the following manifest:

1. Apply the manifest:

```sh kubectl create -f https://k8s.io/examples/controllers/job-pod-failure-policy-ignore.yaml ```

1. Run this command to check the `nodeName` the Pod is scheduled to:

```sh nodeName=$(kubectl get pods -l job-name=job-pod-failure-policy-ignore -o jsonpath='{.items[0].spec.nodeName}') ```

1. Drain the node to evict the Pod be …(trimmed)