Operating etcd clusters for Kubernetes [page]deterministic

tasks

[definition:etcd]

Before you follow steps in this page to deploy, manage, back up or restore etcd, you need to understand the typical expectations for operating an etcd cluster. Refer to the [etcd documentation](https://etcd.io/docs/) for more context.

Key details include:

* The minimum recommended etcd versions to run in production are `3.4.29+` and `3.5.11+`.

* etcd is a leader-based distributed system. Ensure that the leader periodically send heartbeats on time to all followers to keep the cluster stable.

* You should run etcd as a cluster with an odd number of members.

* Aim to ensure that no resource starvation occurs.

Performance and stability of the cluster is sensitive to network and disk I/O. Any resource starvation can lead to heartbeat timeout, causing instability of the cluster. An unstable etcd indicates that no leader is elected. Under such circumstances, a cluster cannot make any changes to its current state, which implies no new pods can be scheduled.

### Resource requirements for etcd

Operating etcd with limited resources is suitable only for testing purposes. For deploying in production, advanced hardware configuration is required. Before deploying etcd in production, see [resource requirement reference](https://etcd.io/docs/current/op-guide/hardware/#example-hardware-configurations).

Keeping etcd clusters stable is critical to the stability of Kubernetes clusters. Therefore, run etcd clusters on dedicated machines or isolated environments for [guaranteed resource requirements](https://etcd.io/docs/current/op-guide/hardware/).

### Tools

Depending on which specific outcome you're working on, you will need the `etcdctl` tool or the `etcdutl` tool (you may need both).

## Understanding etcdctl and etcdutl

`etcdctl` and `etcdutl` are command-line tools used to interact with etcd clusters, but they serve different purposes:

`etcdctl`: This is the primary command-line client for interacting with etcd over a network. It is used for day-to-day operations such as managing keys and values, administering the cluster, checking health, and more.

`etcdutl`: This is an administration utility designed to operate directly on etcd data files, including migrating data between etcd versions, defragmenting the database, restoring snapshots, and validating data consistency. For network operations, `etcdctl` should be used.

For more information on `etcdutl`, you can refer to the [etcd recovery documentation](https://etcd.io/docs/v3.5/op-guide/recovery/).

## Starting etcd clusters

This section covers starting a single-node and multi-node etcd cluster.

This guide assumes that `etcd` is already installed.

### Single-node etcd cluster

Use a single-node etcd cluster only for testing purposes.

1. Run the following:

```sh etcd --listen-client-urls=http://$PRIVATE_IP:2379 \ --advertise-client-urls=http://$PRIVATE_IP:2379 ```

2. Start the Kubernetes API server with the flag `--etcd-servers=$PRIVATE_IP:2379`.

Make sure `PRIVATE_IP` is set to your etcd client IP.

### Multi-node etcd cluster

For durability and high availability, run etcd as a multi-node cluster in production and back it up periodically. A five-member cluster is recommended in production. For more information, see [FAQ documentation](https://etcd.io/docs/current/faq/#what-is-failure-tolerance).

As you're using Kubernetes, you have the option to run etcd as a container inside one or more Pods. The `kubeadm` tool sets up etcd [static pods](#gloss:static-pod) by default, or you can deploy a [separate cluster](/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/) and instruct kubeadm to use that etcd cluster as the control plane's backing store.

You configure an etcd cluster either by static member information or by dynamic discovery. For more information on clustering, see [etcd clustering documentation](https://etcd.io/docs/current/op-guide/clustering/).

For an example, consider …(trimmed)