Metrics For Kubernetes System Components [page]deterministic

System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts.

Kubernetes components emit metrics in [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/). This format is structured plain text, designed so that people and machines can both read it.

## Metrics in Kubernetes

In most cases metrics are available on `/metrics` endpoint of the HTTP server. For components that don't expose endpoint by default, it can be enabled using `--bind-address` flag.

Examples of those components:

* [kube-controller-manager](#gloss:kube-controller-manager) * [kube-proxy](#gloss:kube-proxy) * [kube-apiserver](#gloss:kube-apiserver) * [kube-scheduler](#gloss:kube-scheduler) * [kubelet](#gloss:kubelet)

In a production environment you may want to configure [Prometheus Server](https://prometheus.io/) or some other metrics scraper to periodically gather these metrics and make them available in some kind of time series database.

Note that [kubelet](#gloss:kubelet) also exposes metrics in `/metrics/cadvisor`, `/metrics/resource` and `/metrics/probes` endpoints. Those metrics do not have the same lifecycle.

If your cluster uses [RBAC](#gloss:rbac), reading metrics requires authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing `/metrics`. For example:

```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - nonResourceURLs: - "/metrics" verbs: - get ```

## Metric lifecycle

Alpha metric → Beta metric → Stable metric → Deprecated metric → Hidden metric → Deleted metric

Alpha metrics have no stability guarantees. These metrics can be modified or deleted at any time.

Beta metrics observe a looser API contract than its stable counterparts. No labels can be removed from beta metrics during their lifetime, however, labels can be added while the metric is in the beta stage.

Stable metrics are guaranteed to not change. This means:

* A stable metric without a deprecated signature will not be deleted or renamed * A stable metric's type will not be modified

Deprecated metrics are slated for deletion, but are still available for use. These metrics include an annotation about the version in which they became deprecated.

For example:

* Before deprecation

``` # HELP some_counter this counts things # TYPE some_counter counter some_counter 0 ```

* After deprecation

``` # HELP some_counter (Deprecated since 1.15.0) this counts things # TYPE some_counter counter some_counter 0 ```

Hidden metrics are no longer published for scraping, but are still available for use. A deprecated metric becomes a hidden metric after a period of time, based on its stability level: * STABLE metrics become hidden after a minimum of 3 releases or 9 months, whichever is longer. * BETA metrics become hidden after a minimum of 1 release or 4 months, whichever is longer. * ALPHA metrics can be hidden or removed in the same release in which they are deprecated.

To use a hidden metric, you must enable it. For more details, refer to the [Show hidden metrics](#show-hidden-metrics) section.

Deleted metrics are no longer published and cannot be used.

## Show hidden metrics

As described above, admins can enable hidden metrics through a command-line flag on a specific binary. This intends to be used as an escape hatch for admins if they missed the migration of the metrics deprecated in the last release.

The flag `show-hidden-metrics-for-version` takes a version for which you want to show metrics deprecated in that release. The version is expressed as x.y, where x is the major version, y is the minor version. The patch version is not needed even though a metrics can be deprecated in a patch release, the reason for that is the metrics deprecation policy runs against the minor release.

The flag can only take the previous minor ver …(trimmed)