Recently I received a question about how to limit the available load balancer addresses in a vSphere /w Tanzu namespace. Background of this question is a scenario where you don’t want your pool of available external IP addresses to be consumed by a single vSphere namespace.
About vSphere namespaces
As you know vSphere with Tanzu provides Kubernetes capabilities through the Workload Management option in the vSphere WebClient. After enabling Workload Management you can create (vSphere) namespaces in the Webclient (or in same scenarios through the command line using the Tanzu Namespace Service). Access to a vSphere Namespace can be provided to a development team, after which they have to ability to deploy Tanzu Kubernetes Clusters (TKCs) in the vSphere namespace they have access to.
By default people how have access to a vSphere namespace are free to consume all available resources. This means then can deploy as many Kubernetes clusters as they want and in theory they can consume all available resources (CPU, memory, storage). Luckily there are options to limit resource consumption through the Capacity and Usage widget in the vSphere namespace:
On top of this there’s also an option to limit consumption of Kubernetes objects:
This option is available under configure -> object limits in the vSphere namespace.
The screendump in this article is coming from an AVI backed vSphere with Tanzu installation, an NSX-T backed installation will include additional Kubernetes objects such as vSphere Pods, Deployments, Replica Sets, Jobs etc. The full list of available objects in case of NSX-T is available here. Important is to understand that the objects that are referred to, are objects that are available in the namespace on the Supervisor cluster.
How to use Kubernetes Object Limits to limit available load balancer IP addresses?
So now is the question: how can we leverage these objects to limit available load balancer addresses in a vSphere Namespace and thus on the TKG cluster(s) running in this namespace? In case an application is requesting a load balancer address in a TKG cluster running in a vSphere namespace, the request will be redirected to the supervisor cluster. The supervisor cluster will communicatie with HA Proxy/AVI/NSX-T and request the load balancing address. The load balancing address will be provided back to the TKG cluster and can be used by the application. We have to take into account that each TKG cluster will initially consume 1 IP address to get the control plane up and running. The consumed load balancer (and ingress) addresses in a namespace are counted towards the “network/services” limit as shown in the screendump, it doesn’t if these are control plane or application level load balancing addresses.
So let’s configure a namespace with a “network/services” limit of 2, that means we can deploy a TKG cluster (consuming one IP address) and one application also consuming one IP address.
In this example I’ve deployed a cluster called TKG01. I’ve created two namespaces (nginx01 and nginx02) that are hosting a basic nginx webserver. Let’s check the available PODs and services:
As you can see, the service in the nginx01 namespace consumes a load balancing address while the second load balancing request shows status “pending”.
Let’s see what’s going on:
So the load balancer request is stuck and no (available) IP address is found/available.
Let’s check the supervisor cluster of this environment, to see what’s going on in my vSphere Namespace called “ns01”:
As you can see two IP address (network-services) are consumed on the supervisor cluster in the ns01 namespace. Because no additional IP addresses (network services) are available, no extra load balancers can be created in my TKG01 cluster.
We can verify the limit configuration of the namespace:
As you can see “count/services” show a limit of 2, the used value is also 2.
Now let’s change the count/services value to 3 in the vSphere WebClient, and see what happens. A third IP address is made available:
After redeploying the nginx service in the nginx02 namespace, a load balancer address will be provided and the webserver will be made available externally:
And the NS01 vSphere namespace is also showing this third (load balancing) IP address:
This concludes my article, I hope this was helpful. Feel free to leave a comment below.
