Coarse Parallel Processing Using a Work Queue [page]deterministic

tasks

In this example, you will run a Kubernetes Job with multiple parallel worker processes.

In this example, as each pod is created, it picks up one unit of work from a task queue, completes it, deletes it from the queue, and exits.

Here is an overview of the steps in this example:

1. Start a message queue service. In this example, you use RabbitMQ, but you could use another one. In practice you would set up a message queue service once and reuse it for many jobs. 1. Create a queue, and fill it with messages. Each message represents one task to be done. In this example, a message is an integer that we will do a lengthy computation on. 1. Start a Job that works on tasks from the queue. The Job starts several pods. Each pod takes one task from the message queue, processes it, and exits.

You should already be familiar with the basic, non-parallel, use of [Job](/docs/concepts/workloads/controllers/job/).

You will need a container image registry where you can upload images to run in your cluster.

This task example also assumes that you have Docker installed locally.

## Starting a message queue service

This example uses RabbitMQ, however, you can adapt the example to use another AMQP-type message service.

In practice you could set up a message queue service once in a cluster and reuse it for many jobs, as well as for long-running services.

Start RabbitMQ as follows:

```shell # make a Service for the StatefulSet to use kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq/rabbitmq-service.yaml ``` ``` service "rabbitmq-service" created ```

```shell kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq/rabbitmq-statefulset.yaml ``` ``` statefulset "rabbitmq" created ```

## Testing the message queue service

Now, we can experiment with accessing the message queue. We will create a temporary interactive pod, install some tools on it, and experiment with queues.

First create a temporary interactive Pod.

```shell # Create a temporary interactive container kubectl run -i --tty temp --image ubuntu:22.04 ``` ``` Waiting for pod default/temp-loe07 to be running, status is Pending, pod ready: false ... [ previous line repeats several times .. hit return when it stops ] ... ```

Note that your pod name and command prompt will be different.

Next install the `amqp-tools` so you can work with message queues. The next commands show what you need to run inside the interactive shell in that Pod:

```shell apt-get update && apt-get install -y curl ca-certificates amqp-tools python3 dnsutils ```

Later, you will make a container image that includes these packages.

Next, you will check that you can discover the Service for RabbitMQ:

``` # Run these commands inside the Pod # Note the rabbitmq-service has a DNS name, provided by Kubernetes: nslookup rabbitmq-service ``` ``` Server: 10.0.0.10 Address: 10.0.0.10#53

Name: rabbitmq-service.default.svc.cluster.local Address: 10.0.147.152 ``` (the IP addresses will vary)

If the kube-dns addon is not set up correctly, the previous step may not work for you. You can also find the IP address for that Service in an environment variable:

```shell # run this check inside the Pod env | grep RABBITMQ_SERVICE | grep HOST ``` ``` RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152 ``` (the IP address will vary)

Next you will verify that you can create a queue, and publish and consume messages.

```shell # Run these commands inside the Pod # In the next line, rabbitmq-service is the hostname where the rabbitmq-service # can be reached. 5672 is the standard port for rabbitmq. export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672 # If you could not resolve "rabbitmq-service" in the previous step, # then use this command instead: BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672

# Now create a queue:

/usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d ``` ``` foo ```

Publish one message to the q …(trimmed)