Logging Architecture [page]deterministic

concepts

Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams.

However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution.

For example, you may want to access your application's logs if a container crashes, a pod gets evicted, or a node dies.

In a cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called [cluster-level logging](#cluster-level-logging-architectures).

Cluster-level logging architectures require a separate backend to store, analyze, and query logs. Kubernetes does not provide a native storage solution for log data. Instead, there are many logging solutions that integrate with Kubernetes. The following sections describe how to handle and store logs on nodes.

## Pod and container logs {#basic-logging-in-kubernetes}

Kubernetes captures logs from each container in a running Pod.

This example uses a manifest for a `Pod` with a container that writes text to the standard output stream, once per second.

To run this pod, use the following command:

```shell kubectl apply -f https://k8s.io/examples/debug/counter-pod.yaml ```

The output is:

```console pod/counter created ```

To fetch the logs, use the `kubectl logs` command, as follows:

```shell kubectl logs counter ```

The output is similar to:

```console 0: Fri Apr 1 11:42:23 UTC 2022 1: Fri Apr 1 11:42:24 UTC 2022 2: Fri Apr 1 11:42:25 UTC 2022 ```

You can use `kubectl logs --previous` to retrieve logs from a previous instantiation of a container. If your pod has multiple containers, specify which container's logs you want to access by appending a container name to the command, with a `-c` flag, like so:

```shell kubectl logs counter -c count ```

### Container log streams

As an alpha feature, the kubelet can split out the logs from the two standard streams produced by a container: [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)) and [standard error](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_(stderr)). To use this behavior, you must enable the `PodLogsQuerySplitStreams` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). With that feature gate enabled, Kubernetes allows access to these log streams directly via the Pod API. You can fetch a specific stream by specifying the stream name (either `Stdout` or `Stderr`), using the `stream` query string. You must have access to read the `log` subresource of that Pod.

To demonstrate this feature, you can create a Pod that periodically writes text to both the standard output and error stream.

To run this pod, use the following command:

```shell kubectl apply -f https://k8s.io/examples/debug/counter-pod-err.yaml ```

To fetch only the stderr log stream, you can run:

```shell kubectl get --raw "/api/v1/namespaces/default/pods/counter-err/log?stream=Stderr" ```

See the [`kubectl logs` documentation](/docs/reference/generated/kubectl/kubectl-commands#logs) for more details.

### How nodes handle container logs

![Node level logging](/images/docs/user-guide/logging/logging-node-level.png)

A container runtime handles and redirects any output generated to a containerized application's `stdout` and `stderr` streams. Different container runtimes implement this in different ways; however, the integration with the kubelet is standardized as the _CRI logging format_.

By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers …(trimmed)