How to connect vROps to TKG running in vSphere with Kubernetes

With the vRealize Operations Management Pack for Container Monitoring you get a wealth on information to monitor what’s happening inside your Kubernetes clusters.

You can use the management pack to monitor your Tanzu Kubernetes Grid (TKG) clusters, including TKG Service clusters – that are TKG clusters that are running in vSphere with Kubernetes. To learn more about the different TKG offerings, read this article.

There are various blogposts on how to connect vROps to a (stand-alone) TKG cluster, but I had some difficulties on how to connect vROps to TKG Service. This has to do with the difference approach on how a TKG Service cluster is deployed and managed. In vSphere with Kubernetes the supervisor cluster is used to manage your TKG cluster, this required another approach to retrieve the client-certificate-data en key.

From now on I will talk about a TKG cluster, this is the K8S cluster (TKG Service cluster) running in vSphere with Kubernetes.

After installing the Management Pack for Container Monitoring in vROps, you have to deploy cAdvisor to your TKG cluster to collect additional metrics and properties of Kubernetes objects by running vAdvisor in a sidecar pod. After cAdvisor is deployed, you have to retrieve certificate data.

After you’ve enabled workload management in vSphere and created and configured a namespace, it’s time to deploy a TKG cluster.

Logon to the supervisor cluster:

kubectl vsphere login --server 172.16.200.33 -u devops@viktorious.local --insecure-skip-tls-verify

Switch to the namespace/context that will have the TKG cluster deployed (not strictly necessary because the context is also defined in the TKG deployment yaml):

kubectl config use-context viktorious01

Deploy a TKG cluster:

kubectl apply -f tkg1.yaml

This article provides more details on how to deploy a TKG cluster in vSphere with Kubernetes.

Retrieve TKG secrets

After the TKG cluster is successfully deployed, we will need to retrieve the certificate-authority-data, client-certificate-date and client-key-date. We will need this information to connect vROps to this TKG cluster. Because (by default) we don’t have access to the TKG node command line, we have to retrieve this information from the SuperVisor cluster through kubectl. Each TKG cluster in vSphere with Kubernetes has a list of secrets available that store tokens, keys and password for operating TKG.

Retrieve available secrets on the SuperVisor cluster:

kubectl get secrets

The secret you will need is TKG-CLUSTER-NAME-kubeconfig. To retrieve the contents of this secret use the following command:

kubectl get secret tkg-cluster-1-kubeconfig -o jsonpath='{.data.value}' | base64 -d > tkg-cluster-kubeconfig-admin

In this example tkg-cluster-1 is the name of the TKG cluster. The contents of the secret is written to a file named tkg-cluster-kubeconfig-admin. Open this file and retrieve the information behind certificate-authority-data, client-certificate-data and client-key-date. Also record the IP address and port number behind “server:” (you will need this information later).

You can also retrieve the IP address and port number of the cluster using:

kubectl get service

And look for the CLUSTERNAME-control-plane-service. More details on retrieving TKG secrets is published here.

Deploy cAdvisor

Of course we still have to deploy the cAdvisor daemonset. For this logon to (context of) the TKG cluster with the following command:

kubectl vsphere login --server 172.16.200.33 --tanzu-kubernetes-cluster-name tkg-cluster-1 --tanzu-kubernetes-cluster-namespace viktorious01 --vsphere-username devops@viktorious.local --insecure-skip-tls-verify

Deploy the cAdvisor daemonset to the TKG cluster.

kubectl apply -f cadvisor.yaml

An example of cAdvisor.yaml is available here.

Check if the pods are running:

kubectl get pods -A

Configure vROps

Now it’s time to configure vRealize Operations. Go to Administration->Other Accounts and choose Add account.

Note: don’t forget to configure the vCenter Server FQDN. This will map Kubernetes/TKG nodes to their respective virtual machines.

Use credential to provide the certificate-authority-data, client-certificate-data and client-key-date:

Click Validate Connection to test connectivity and click save to start monitoring.

That’s it, now you’re ready to start monitoring your TKG Service with vRealize Operations!