In this post I will share my experiences and some tips & tricks for deploying TKG Multi-Cloud (unofficially called TKGm) using the Service Installer for VMware Tanzu (SIVT). The Service Installer is a free tool available on the VMware Marketplace that will automate and simplify the initial deployment of any Tanzu Kubernetes Grid (TKG) deployment, including NSX-ALB/AVI (if applicable). SIVT can also connect your TKG installation to Tanzu Mission Control and Tanzu Observability(Aria Operations for Applications).
The Service Installer can be used for different deployment scenarios, including:
- Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on VMware Cloud
- Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on vSphere with NSX-T
- Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on vSphere running Distributed Switch (vDS)
- vSphere with Tanzu (aka TKGs or TKG service) running vSphere Distributed Switch (vDS)
- Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on AWS (air-gapped and internet-connected)
- Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on Azure
The SIVT is following the reference design for Tanzu for Kubernetes Operations as published here. An interesting use-case for the SIVT is deploy Tanzu in an automated way to Edge locations. Although it’s valid use-case, SIVH is currently not following the Edge Reference Architecture as published here (but is following the TKO reference architecture).
Deploy TKGm 1.6 to a single host vSphere 7/8 environment with VDS
In this article I will specifically focus on “Tanzu Kubernetes Grid (multi-cloud, aka TKGm) on vSphere running vSphere Distributed Switch (VDS)” architecture. In this article I share my experiences based on deploying TKGm 1.6 on vSphere. Note that currently only vSphere 7 is supported for TKGm 1.6. I deployed TKGm 1.6 to vSphere 8 and didn’t encounter any (vSphere 8 related) issues. In this architecture NSX Advanced Load Balancer (NSX-ALB, aka AVI) is used for load balancing. I am deploying TKGm to my homelab, that only has one ESXi host available.
Important: Deployment of TKGm to a single host with SIVT requires a minor change to the scripts available on the SIVT appliance. The default script assume a shared datastore to deploy the AVI service engines. In a single host environment you don’t have any shared datastores (shared across hosts), this will result in an error during deployment. More details on this and a solution is available further down below.
Prerequisites to successfully deploy TKGm with DVS and AVI are:
- A vSphere 7 (or 8) environment – vSphere 8 officially not yet supported.
- You will need 6 pre-configured portgroups that should all be able to route to each other. DHCP should be configured of two of these networks. On top of that you will also need to reserve some IP block in the VIP/data networks. Full details are available here.
- For automated download of the required software components (TKGm, NSX-ALB) a Cloud Service Portal API token is required.
- DNS name resolution should be in place for NSX-ALB and Harbor.
- You will need to download the Service Installer OVA from the VMware marketplace.
The main process to deploy TKGm on vSphere running VDS using the Service Installer for VMware Tanzu (SIVT):
- Get the prerequisites in place – configure your network.
- Deploy the SIVT appliance.
- Configure a deployment JSON using the SIVT appliance and save/download the configuration JSON.
- Start the deployment process using the arcas command on the SIVT appliance.
Prerequisites: configure your network
In this scenario I created six portgroups/VLANs, following the architecture as illustrated here:
Some remarks:
The documentation shows 6 networks in the architecture diagram, however in the list of pre-requisites only 5 networks/portgroups are mentioned. The TKG Cluster VIP network is missing from this list, please note that this network is required and should be pre-configured. No DHCP is required on this network.
The documentation has been updated and now reflects 6 networks.- I’m using a Standard Switch in this setup, this works fine in my single host setup. Officially a Distributed Switch (vDS) is required.
To keep things simple, I used /24 subnets for all the networks: 172.16.201, 202, 203, 204, 205, 206.0/24. Both VLAN 202 and 206 have a DHCP server configured, these networks will have the TKG node management IPs connected to them. All networks are routable. The DHCP server should advertise a NTP server. Full networking requirements are here.
On top of this AVI/NSX-ALB will also create an “AVI internal” portgroup. This portgroup is used to connect unused SE network adapter to and doesn’t need connectivity to the outside world.
Deploy the SIVT appliance
Deploying the SIVT appliance is just like deploying any other OVA. Just follow the wizard and submit the required information.
Creating the configuration JSON for your TKGm setup
Access the SIVT appliance at <SIVT-ADDRESS>:8888. And choose the option “Tanzu Kubernetes Grid Multi-Cloud on VMware vSphere with DVS”. The JSON configuration wizard consists of 13 steps, note that not all steps are mandatory (depending on your requirements). Step 1, 2, 3, 6, 8 and 9 are mandatory – the other steps depend on your configuration:
- Do you want to use and include a Tanzu Mission Control configuration?
- Do you want to create a Workload Cluster and/or Services Cluster?
- Do you want to configure identity management using LDAPS or OIDC?
- …
Most of the options are pretty straight forward, however for the automatic download of the required files you would need a VMware Marketplace account + token. This CSP Marketplace API token can be retrieved from your CSP account here: https://console.cloud.vmware.com/csp/gateway/portal/#/user/tokens. Make sure you’ve selected the correct organization and generate an API token for the marketplace:
Add this token top step 3 of JSON configuration wizard. I’ve created a sample configuration JSON that is available for download on GitHub. You can upload this JSON to your SIVT appliance so explore the sample configuration.
After you’ve completed the JSON configuration wizard you can save the JSON to the VM and download the file (choose view and then download to download the file). 
The file is saved to /opt/vmware/arcas/src/vsphere-dvs-tkgm.json on your SIVT appliance.
Deployment proces using the arcas command
The arcas command will execute the actual deployment of your environment. Because of the lack of shared datastores in a single host enviroment, I’ve made a few changes to the file /opt/vmware/arcas/src/vsphere/managementConfig/vsphere_management_config.py available on the SIVT:
On line 1846 and 2028, change
"vcenter_datastore_mode": "VCENTER_DATASTORE_SHARED",
to
“vcenter_datastore_mode”: “VCENTER_DATASTORE_ANY”,
Remove
"vcenter_datastores_include": True,
from line 1845 and 2047. Also remove
"vcenter_datastores": [{
"datastore_name": dataStore
}],
from lines 2003-2005 and 2200-2202.
Now you can start the deployment using the arcas command. Basic deployment of AVI + TKGm Management cluster:
arcas --env vsphere --file /opt/vmware/arcas/src/vsphere-dvs-tkgm.json --avi_configuration --tkg_mgmt_configuration
If you want to deploy AVI + TKGm Management & Workload cluster:
arcas --env vsphere --file /opt/vmware/arcas/src/vsphere-dvs-tkgm.json --avi_configuration --tkg_mgmt_configuration --workload_preconfig --workload_deploy
If you also want to deploy a TKGm Services cluster, your command will look like:
arcas --env vsphere --file /opt/vmware/arcas/src/vsphere-dvs-tkgm.json --avi_configuration --tkg_mgmt_configuration --shared_service_configuration --workload_preconfig --workload_deploy
You can add –deploy_extensions to the command to deploy extensions.
Note: If you are doing a phased installation, it’s perfectly fine to first run the arcas command to install the management cluster and a later stage re-run the arcas command and add the services cluster parameter. In the final stage you can then deploy the workload cluster.
After you’ve started the deployment I recommend to monitor the process using:
tail -f /var/log/server/arcas.log
Final thoughts
If everything goes as expected, you will get a freshly installed TKGm environment configured according to the settings you supplied in the SIVT wizard and/or configuration JSON.
