Fine Parallel Processing Using a Work Queue [page]deterministic

In this example, you will run a Kubernetes Job that runs multiple parallel tasks as worker processes, each running as a separate Pod.

In this example, as each pod is created, it picks up one unit of work from a task queue, processes it, and repeats until the end of the queue is reached.

Here is an overview of the steps in this example:

1. Start a storage service to hold the work queue. In this example, you will use Redis to store work items. In the [previous example](/docs/tasks/job/coarse-parallel-processing-work-queue), you used RabbitMQ. In this example, you will use Redis and a custom work-queue client library; this is because AMQP does not provide a good way for clients to detect when a finite-length work queue is empty. In practice you would set up a store such as Redis once and reuse it for the work queues of many jobs, and other things. 1. Create a queue, and fill it with messages. Each message represents one task to be done. In this example, a message is an integer that we will do a lengthy computation on. 1. Start a Job that works on tasks from the queue. The Job starts several pods. Each pod takes one task from the message queue, processes it, and repeats until the end of the queue is reached.

##

You will need a container image registry where you can upload images to run in your cluster. The example uses [Docker Hub](https://hub.docker.com/), but you could adapt it to a different container image registry.

This task example also assumes that you have Docker installed locally. You use Docker to build container images.

Be familiar with the basic, non-parallel, use of [Job](/docs/concepts/workloads/controllers/job/).

## Starting Redis

For this example, for simplicity, you will start a single instance of Redis. See the [Redis Example](https://github.com/kubernetes/examples/tree/master/web/guestbook/) for an example of deploying Redis scalably and redundantly.

You could also download the following files directly:

• [`redis-pod.yaml`](/examples/application/job/redis/redis-pod.yaml)
• [`redis-service.yaml`](/examples/application/job/redis/redis-service.yaml)
• [`Dockerfile`](/examples/application/job/redis/Dockerfile)
• [`job.yaml`](/examples/application/job/redis/job.yaml)
• [`rediswq.py`](/examples/application/job/redis/rediswq.py)
• [`worker.py`](/examples/application/job/redis/worker.py)

To start a single instance of Redis, you need to create the redis pod and redis service:

```shell kubectl apply -f https://k8s.io/examples/application/job/redis/redis-pod.yaml kubectl apply -f https://k8s.io/examples/application/job/redis/redis-service.yaml ```

## Filling the queue with tasks

Now let's fill the queue with some "tasks". In this example, the tasks are strings to be printed.

Start a temporary interactive pod for running the Redis CLI.

```shell kubectl run -i --tty temp --image redis --command "/bin/sh" ``` ``` Waiting for pod default/redis2-c7h78 to be running, status is Pending, pod ready: false Hit enter for command prompt ```

Now hit enter, start the Redis CLI, and create a list with some work items in it.

```shell redis-cli -h redis ``` ```console redis:6379> rpush job2 "apple" (integer) 1 redis:6379> rpush job2 "banana" (integer) 2 redis:6379> rpush job2 "cherry" (integer) 3 redis:6379> rpush job2 "date" (integer) 4 redis:6379> rpush job2 "fig" (integer) 5 redis:6379> rpush job2 "grape" (integer) 6 redis:6379> rpush job2 "lemon" (integer) 7 redis:6379> rpush job2 "melon" (integer) 8 redis:6379> rpush job2 "orange" (integer) 9 redis:6379> lrange job2 0 -1 1) "apple" 2) "banana" 3) "cherry" 4) "date" 5) "fig" 6) "grape" 7) "lemon" 8) "melon" 9) "orange" ```

So, the list with key `job2` will be the work queue.

Note: if you do not have Kube DNS setup correctly, you may need to change the first step of the above block to `redis-cli -h $REDIS_SERVICE_HOST`.

## Create a container image {#create-an-image}

Now you are ready to create an image that will …(trimmed)