How to deploy vSphere 7 with Kubernetes on a single not-nested ESXi node

In a previous article I explained how to deploy vSphere with Kubernetes on a nested two node ESXi environment. Because the deployment wizard doesn’t check if you have the required number of ESXi hosts available you can even choose to deploy vSphere with Kubernetes on a single host. William Lam already posted an article on this: his article explains a nested single node setup. Although this works fine, you might face a high (CPU) resource demand if your homelab server is equiped with a not too powerful CPU because of the nested configuration.

So I was wondering, can you also deploy vSphere with Kubernetes on a single “physical” ESXi node? This is possible, however you need a slightly different NSX setup. My former colleague Ronald de Jong pointed me out how to configure a collapsed compute and edge cluster.

Note: Of course a single ESXi node setup is not supported configuration for vSphere with Kubernetes and only intended for homelab/testing setups.

So, what does this look like? I’ve detailed required networking architecture in the following diagram:

We’ve got three distributed port groups here that are required for this setup:

• DVS-VLAN178 – The management VLAN, connected to my domain controller (DC01), VCSA (vcenter) and NSX appliance. This port group is also connected to the management interface of my NSX Edge Transport Node (nsx-edge01), and last but not least the ESXi TEP is also connected to this VLAN.
• DVS-VLAN179 – is used to connect the NSX Edge Transport Node TEP and will carry all overlay segments.
• DVS-VLAN-TRUNK – is a virtual trunk port for the NSX Edge Transport Node and will carry all VLAN segments.

The essential thing here is that the TEP of your ESXi host is in a different VLAN than the TEP of the Edge Transport Node. This also means that encapsulated Geneve traffic must be routed between these VLANs. If you connect both TEPs to the same VLAN, you get some unexpected behaviour and the setup won’t work.

So what’s the config like for the NSX Transport Node? To use vSphere with Kubernetes you would need (at least) two transports zones (TZ): an overlay TZ and a VLAN TZ. Normally you would add both TZs to the same N-VDS that is running on the Edge Transport Node. In this scenario you have to create to different N-VDSes, one that’s connected to the overlay TZ and one for the VLAN TZ.

This N-VDS is connected to the DVS-VLAN179 portgroup.

This portgroup is connected to the DVS-VLAN-TRUNK portgroup (that is configured as a VLAN trunk on the dvSwitch level). You have to deploy both n-NDSes to the same Edge Transport Node.

Don’t forget to configure MTU 1600 on both the dvSwitch and the router that will route the Geneve/TEP traffic. Continue configuring NSX-T as detailed in the configuration guide.

Now you’ve setup NSX-T on this standalone ESXi host, it’s time to configure Workload Management and enable vSphere with Kubernetes.

Just setup the environment as you would normally do, take a look at my previous post on how to setup Workload Management in a 2 Node Lab Environment. The setup of this “physical” setup takes much less time, and the average load on my ESXi lab server is 30% lower compared to a nested setup. Also don’t forget to tweak the number of VMs in the SuperVisor cluster before you enable Workload Management: normally there are 3 VMs but you can bring this down to 2 by changing the settings in /etc/vmware/wcp/wcpsvc.yaml on your vCenter Server. More details are available in my previous post. You can also use a local storage volume for this setup if you would like to.

At the end (and after configuring a first Namespace) your environment will look this:

I hope this was helpful, good luck with setting up your own vSphere with Kubernetes lab environment.