Migrate Kubernetes Objects Using Storage Version Migration [page]deterministic

Kubernetes relies on API data being actively re-written, to support some maintenance activities related to at rest storage. Two prominent examples are the versioned schema of stored resources (that is, the preferred storage schema changing from v1 to v2 for a given resource) and encryption at rest (that is, rewriting stale data based on a change in how the data should be encrypted).

Running storage version migrations allows for the assurance that all objects for a Resource have been migrated off of a stale storage version. The requirements to running a storage migration is ensuring that the Resource has an integer resource version. All Kubernetes Resources and CRDs are ensured to have this property, but migration will fail if this is not the case, for instance with aggregated APIs.

##

Install [`kubectl`](/docs/tasks/tools/#kubectl).

Ensure that your cluster has the `StorageVersionMigrator` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/#StorageVersionMigrator) enabled. You will need control plane administrator access to make that change.

Enable storage version migration REST API by setting runtime config `storagemigration.k8s.io/v1beta1` to `true` for the API server. For more information on how to do that, read [enable or disable a Kubernetes API](/docs/tasks/administer-cluster/enable-disable-api/).

## Re-encrypt Kubernetes secrets using storage version migration

• To begin with, [configure KMS provider](/docs/tasks/administer-cluster/kms-provider/) to encrypt data at rest in etcd using following encryption configuration.

```yaml kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== ```

Make sure to enable automatic reload of encryption configuration file by setting `--encryption-provider-config-automatic-reload` to true.

• Create a Secret using kubectl.

```shell kubectl create secret generic my-secret --from-literal=key1=supersecret ```

• [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted) the serialized data for that Secret object is prefixed with `k8s:enc:aescbc:v1:key1`.

• Update the encryption configuration file as follows to rotate the encryption key.

```yaml kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key2 secret: c2VjcmV0IGlzIHNlY3VyZSwgaXMgaXQ/ - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== ```

• To ensure that previously created secret `my-secret` is re-encrypted with new key `key2`, you will use _Storage Version Migration_.

• Create a StorageVersionMigration manifest named `migrate-secret.yaml` as follows:

```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1beta1 metadata: name: secrets-migration spec: resource: group: "" resource: secrets ```

Create the object using `kubectl` as follows:

```shell kubectl apply -f migrate-secret.yaml ```

• Monitor migration of Secrets by checking the `.status` of the StorageVersionMigration. A successful migration should have its `Succeeded` condition set to true. Get the StorageVersionMigration object as follows:

```shell kubectl wait --for=condition=Succeeded storageversionmigration.storagemigration.k8s.io/secrets-migration ```

The output is similar to:

```yaml kind: StorageVersionMigration apiVersion: storagemigration.k8s.io/v1beta1 metadata: name: secrets-migration uid: 628f6922-a9cb-4514-b076-12d3c178967c resourceVersion: "90" creationTimestamp: "2024-03-12T20:29:45Z" spec: resource: group: "" resource: secrets status: conditions: - type: Running status: "False" lastUpdate …(trimmed)