In a previous post I explained how to configure L4 load balancing with Cilium and a Ubuiqiti Edge Router running EdgeOS.

In this post we will dive into two new Kubernetes constructs and see how Cilium implements these:

1. Ingress is an API object that manages external access to services in a cluster, typically HTTP(S). Traffic routing is controlled by rules defined on the Ingress resource. Note that Ingress is frozen and new features are added to Gateway API (next bullet).
2. Gateway API is another way to make network services available using an extensible, role-oriented, protocol-aware configuration mechanism and is the successor to Ingress. Gateway API supports various protocols a.o. HTTP(S) and gRPC.

The examples are more or less based on the examples that are also available on the Cilium website, although I have them my own “touch”. I hope it’s useful for you.

Ingress, TLS and Cert-manager

So, Ingress exposes HTTP(S)routes from outside the cluster to services within the cluster. You can define rules on the Ingress resource to route traffic.

You need to enable the Ingress capability on Cilium before you can use it. I’m using Helm to manage my Cilium installation, so this means I have to update my cilium-values.yaml and add the Ingress capability.

# Enables Ingress - needs nodePort.enable=true or kubeProxyReplacement=true
ingressController:
  enabled: "true"
  loadbalancerMode: "shared"
  service:
    name: cilium-ingress
    type: LoadBalancer
    loadBalancerIP : 172.16.101.100

The loadbalancerMode specifies if you want to use a dedicated (meaning you separate load balancers for different Ingress objects) or shared load balancer for Ingress. In this example I’m using shared mode with a specified IP address being 172.16.101.100. On top of that the nodePort or Kube Proxy Replacement and L7 proxy need to be enabled, for more details see my previous post.

Update your Cilium configuration:

helm upgrade cilium cilium/cilium --version 1.14.5 -f cilium-values.yaml -n kube-system

and restart Cilium Operator and Agents:

kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Now check your configuration:

cilium config view | grep ingress-controller

You can check there’s an ingress service running in kube-system:

kubectl get svc cilium-ingress -n kube-system

Let’s now deploy two applications, the MicroServices demo coming from Google and the Bookinfo app coming from Istio. I am using the MicroServices demo quite frequently in demos, in this case I’m deploying the app to a namespace called shop. The Bookinfo app is being deployed to the bookinfo namespace. I’ve also setup cert-manager and configured a cluster-issuer that is configured to interact with a route53 domain as described in this blogpost. This cluster-issuer is available on my cluster and is leveraged by Ingress objects to automatically request and inject signed certificates for my endpoints using annotations.

You can find the Ingress definitions for the MicroServices demo here, for the Bookstore it’s available here. Deploy both ingresses to their respective namespaces. The ingress controller runs in kube-system namespace, while the ingress objects run in the shop en bookstore namespace.

Of course you can use Hubble to monitor network flows:

Gateway API, TLS and cert-manager

Let’s see if we can achieve something similar with Gateway API.

Gateway API is an official Kubernetes project focused on L4 and L7 routing in Kubernetes. This project represents the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs. From the outset, it has been designed to be generic, expressive, and role-oriented.

Before we get started with configuration of Gateway API on Cilium, we need to take care of:

• The kube-proxy-replacement must be installed and L7 proxy should be enabled (just as with Ingress).
• The Gateway API CRDs should be installed on your Kubernetes cluster.

Install the Gateway API CRDs with:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Now enable Gateway API on your Cilium installation, for example through a Helm values.yaml.

# Enables Gateway API
gatewayAPI:
enabled: "true"

Update your Cilium configuration:

helm upgrade cilium cilium/cilium --version 1.14.5 -f cilium-values.yaml -n kube-system

and restart Cilium Operator and Agents:

kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Yes, this is the same procedure as with the enablement of Ingress.

So now it’s time to create a few things:

1. Create a new Gateway with two listeners, one listener is configured to listen to another namespace then where the Gateway resides.
2. Create two HTTP routes, one for Bookinfo app and one route for the Shop app. The HTTP route for the Bookinfo app has two matching rules.
3. Define two ReferenceGrants, this is required because the apps are running in different namespaces.

The final setup is something similar to this (image courtesy gateway-api.sigs.k8s.io):

I’ve combined these 3 configuration in one file. The example is providing https access to both the Bookinfo and Shop app. In the Ingress example we’ve used annotations to automatically generate the TLS certificates; although this is also possible with Gateway API and cert-manager – my setup does not meet the specific requirements so I generated the certificates manually (here and here) and put them in the bookinfo and shop namespace.

In this example the gateway + http routes to both applications are deployed to the bookinfo namespace, this namespace also holds the TLS secret. The MicroServices app is deployed in the shop namespace (and holds the TLS secret), this means that one of the http routes points to another namespace. We need to configure a ReferenceGrant for the http route here, and we also need to configure a ReferenceGrant so the secret that holds the TLS certificate can also be accessed.

A ReferenceGrant can be used to enable cross namespace references within Gateway API. In particular, Routes may forward traffic to backends in other namespaces, or Gateways may refer to Secrets in another namespace.

In the past, we’ve seen that forwarding traffic across namespace boundaries is a desired feature, but without a safeguard like ReferenceGrant, vulnerabilities can emerge.

The specification of these two Reference Grants that are configured at the “receiving side” are included in this yaml file.

Applying this file results in a gateway being configured, as well as two http routes and two reference grants.

I hope this was helpful!