At some point, you may encounter the following issue within your Kubernetes cluster:

NAME READY STATUS RESTARTS AGE
pod01 1/2 ImagePullBackOff 0 24s

The “ImagePullBackOff” error is a common occurrence in Kubernetes, signaling that a container in a pod failed to retrieve the necessary image from a registry. To gain more insights into the problem, you can describe the pod:

kubectl describe pod01

This command will provide a detailed breakdown of the issue. In the example output, you might observe:

Error: ImagePullBackOff
Normal Pulling 15s (x2 over 38s) kubelet Pulling image "<SOME-IMAGE>"
Warning Failed 10s (x2 over 32s) kubelet Failed to pull image "<SOME-IMAGE>": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/<SOME-IMAGE>": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/<SOME-IMAGE-WITH-SOME-ID>: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

In this case you’re facing Docker Hub’s pull rate limit for anonymous accounts. This restriction implies that you’ve exceeded the allowed number of pulls within a specific timeframe.

There are various options to solve this issue. In this blog post, we’ll explore one solution by setting up a Proxy Cache for Docker Hub using Harbor. Additionally, we’ll guide you through the installation and configuration of the Harbor Container Webhook. This webhook will automatically redirect any Docker Hub image pull requests to the configured Proxy Cache within your Harbor registry.

While alternative solutions exist, the advantage of this approach lies in its simplicity—no need to modify any YAML files.

Prerequisites for this setup include:

• An instance of Harbor registry
• Installation of the Harbor Container Webhook on your Kubernetes cluster(s)

Setup Proxy Cache on Harbor

To setup Proxy Cache on Harbor you can follow this procedure.

The first step is to configure a registry endpoint on your Harbor instance, the option is available under Administration->Registries.

Use the Test Connection to verify connectivity. The Access Secret is configured on the Docker Hub Account Settings page.

The next step is to configure a new Project in Harbor that’s linked to the new Registry endpoint:

Now you can test if the Proxy Cache is working as expected:

docker pull <YOUR-HARBOR-URL>/docker_hub/goharbor/harbor-core:dev

Install and configure Harbor Container Webhook

–> Note that cert-manager is prerequisite for installing the Harbor Container Webhook

Now it’s time to install and configure the Harbor Container Webhook. First create a new namespace for the webhook and navigate to the namespace.

kubectl create ns harbor-containerwebhook
kubectl config set-context --current --namespace=harbor-container-webhook

Clone the project to your local system.

git clone https://github.com/indeedeng-alpha/harbor-container-webhook/

Navigate to the /deploy/charts/harbor-container-webhook folder and configure the values.yaml of the helmchart so it reflects your configuration. The changes I’ve made to the values.yaml are:

imagePullSecrets:
  - name: regcred

Note: the initial image that is used by the webhook is also coming from Docker Hub. If you’re already facing Docker Pull Rate Limits you might want to create a secret containing your Docker Hub logon credentials and attach this secret to the values.yaml file.

kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=<DOCKERHUB-ID> --docker-password=<DOCKERHUB-TOKEN> --docker-email=<DOKERHUB-EMAIL>

Create this secret in the harbor-container-webhook namespace.

Now configure the remaining part of the values.yaml file.

## configures the webhook rules, which are evaluated for each image in a pod
rules:
  - name: 'docker.io rewrite rule'
    # image refs must match at least one of the rules, and not match any excludes
    matches:
      - '^docker.io'
#    excludes:
#      # for example, exclude ubuntu from harbor's proxy cache
#      - '^docker.io/(library/)?ubuntu:.*$'
    replace: '<HARBOR-URL>/docker_hub'
    checkUpstream: true
#  - name: 'docker.io ubuntu rewrite rule'
#    # image refs must match at least one of the rules, and not match any excludes
#    matches:
#      - '^docker.io/(library/)?ubuntu:.*$'
#    replace: 'harbor.example.com/ubuntu-proxy'
#    checkUpstream: true # tests if the manifest for the rewritten image exists

Now it’s time to install the webhook:

helm install harbor-container-webhook ./

That’s it! Your Docker Hub image pulls are now being redirected to your Harbor Proxy Cache and you will not face any rate limiting issues! I hope this was helpful.

By the way…did you know you can use Tanzu Mission Control to deploy the webhook to each Kubernetes cluster that is attached to a cluster group in Tanzu Mission Control? A generic configuration sample is provided in this blogpost.