Using sysctls in a Kubernetes Cluster [page]deterministic

tasks

This document describes how to configure and use kernel parameters within a Kubernetes cluster using the [sysctl](#gloss:sysctl) interface.

> Note: Starting from Kubernetes version 1.23, the kubelet supports the use of either `/` or `.` as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as `kernel.shm_rmid_forced` using a period as the separator, or as `kernel/shm_rmid_forced` using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page [sysctl.d(5)](https://man7.org/linux/man-pages/man5/sysctl.d.5.html) from the Linux man-pages project.

> Note: `sysctl` is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.

For some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.

## Listing all Sysctl Parameters

In Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the `/proc/sys/` virtual process file system. The parameters cover various subsystems such as:

kernel (common prefix: `kernel.`)
networking (common prefix: `net.`)
virtual memory (common prefix: `vm.`)
MDADM (common prefix: `dev.`)
More subsystems are described in [Kernel docs](https://www.kernel.org/doc/Documentation/sysctl/README).

To get a list of all parameters, you can run

```shell sudo sysctl -a ```

## Safe and Unsafe Sysctls

Kubernetes classes sysctls as either _safe_ or _unsafe_. In addition to proper namespacing, a _safe_ sysctl must be properly _isolated_ between pods on the same node. This means that setting a _safe_ sysctl for one pod

must not have any influence on any other pod on the node
must not allow to harm the node's health
must not allow to gain CPU or memory resources outside of the resource limits of a pod.

By far, most of the _namespaced_ sysctls are not necessarily considered _safe_. The following sysctls are supported in the _safe_ set:

`kernel.shm_rmid_forced`;
`net.ipv4.ip_local_port_range`;
`net.ipv4.tcp_syncookies`;
`net.ipv4.ping_group_range` (since Kubernetes 1.18);
`net.ipv4.ip_unprivileged_port_start` (since Kubernetes 1.22);
`net.ipv4.ip_local_reserved_ports` (since Kubernetes 1.27, needs kernel 3.16+);
`net.ipv4.tcp_keepalive_time` (since Kubernetes 1.29, needs kernel 4.5+);
`net.ipv4.tcp_fin_timeout` (since Kubernetes 1.29, needs kernel 4.6+);
`net.ipv4.tcp_keepalive_intvl` (since Kubernetes 1.29, needs kernel 4.5+);
`net.ipv4.tcp_keepalive_probes` (since Kubernetes 1.29, needs kernel 4.5+).
`net.ipv4.tcp_rmem` (since Kubernetes 1.32, needs kernel 4.15+).
`net.ipv4.tcp_wmem` (since Kubernetes 1.32, needs kernel 4.15+).

> Note: There are some exceptions to the set of safe sysctls:

The `net.*` sysctls are not allowed with host networking enabled.
The `net.ipv4.tcp_syncookies` sysctl is not namespaced on Linux kernel version 4.5 or lower.

This list will be extended in future Kubernetes versions when the kubelet supports better isolation mechanisms.

### Enabling Unsafe Sysctls

All _safe_ sysctls are enabled by default.

All _unsafe_ sysctls are disabled by default and must be allowed manually by the cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be scheduled, but will fail to launch.

With the warning above in mind, the cluster admin can allow certain _unsafe_ sysctls for very special situations such as high-performance or real-time application tuning. _Unsafe_ sysctls are enabled on a node-by-node basis with a flag of the kubelet; for example:

```shell kubelet --allowed-unsafe-sysctls \ 'kernel.msg*,net.core.somaxconn' ... ```

For [minikube](#gloss:minikube), this can be done via the `extra-config` flag:

```shell minikube start - …(trimmed)