When deploying vCloud Automation Center (vCAC) or vRealize Automation (vRA) you have to deal with a lot of different components that all have their own maximums when it comes to provisioning virtual machines. vCAC/vRA components that are involved in provisioning virtual machines are:
- DEM Orchestrator, monitors DEM Workers, preprocesses workflows to run and schedules the workflows. Does not run/execute the actual workflows.
- DEM Worker, runs workflows. Runs business logic of custom models, interacting with databases and systems. Also manages cloud and physical machines (no virtual machines).
- Proxy Agents, sends commands to- and collects data from vSphere, XenServer and Hyper-V.
- vCenter Orchestrator, responsible for running tenant/customer specific workflows as part of the building machine, machine provisioned and decommission stubs.
Each of these components have their own maximums in regards to concurrent virtual machine provisioning activities. I want to share these maximums in this article and also how to increase these parameters if required.
Note: This information is verified for vCAC 6.1, but not yet checked for vRA 6.2
One DEM Orchestrator is capable of orchestrating many different workflows and up to managing 10.000 virtual machines (or even more). Deploy at least one DEM Orchestrator and one standby DEM Orchestrator. For this component scalability is most of the time not a concern.
Each DEM Worker instance can process 15 concurrent workflows. This also includes scheduled workflows for collecting information about the environment. Excess workflows are queued for execution. Monitor the DEM status under Infrastructure–>Monitoring–>DEM Status, if to many workflows are in the queue, consider to add more DEM workers. Also monitor the virtual machines that are running the DEM worker service, ongoing high CPU load can also be a reason to add more DEM workers.
It’s possible to make DEM Workers location or application specific using the skills functionality. Check chapter four of the vCAC Extensibility Guide for more information:
The configured vCloud Automation Center limit on the number of resource-intensive (typically lengthy) provisioning activities is by default only two. Concurrent activities beyond the configured limit are queued.
By default, vCloud Automation Center limits concurrent virtual provisioning activities for hypervisors that use proxy agents to two per proxy agent. This ensures that the virtualization platform managed by a particular agent never receives enough resource-intensive work items to prevent execution of other items. Plan to carefully test the effects of changing the limit before making any changes. Determining the best limit for your site may require that you investigate work item execution within the virtualization platform as well as workflow activity execution within vCloud Automation Center.
More information is in the vCAC System Administration documentation:
For vCenter Orchestrator server the following maximums are applicable:
- Maximum number of concurrent vCenter Orchestrator workflows: 300
- Connected ESXi instance: 1280
- Connected vCenter Server: 20
- Connected virtual machines: 35000 (15000 per vCO cluster node)
This information is taken from the vSphere Configuration Maximums PDF:
By default vCenter Orchestrator is configured with 2 vCPUs and 4 GB RAM. Consider changing this configuration to for example 4 vCPUs and 8-16 GB of RAM if the load requires this. Carefully monitor CPU and Memory load on your Orchestrator server.
If you see time-outs on your Orchestrator server when retrieving certain objects, you might consider to increase the Java heap size. Read VMware KB 2007423 for additional details.
VMware vCenter Server
VMware is not listing deployment maximums for vCenter Server anymore in the Configuration Maximums PDF. I have found some information on deployment maximums in the documentation for VMware View:
- Maximum concurrent vCenter provisioning operations is 20;
- Maximum concurrent power operations is 50;
This information is taken from the VMware View documentation and thus applies to the View configuration. Although not direct applicable to vCAC and gives you an impression of the capabilities of vCenter Server.
I hope this helps in sizing your vCAC environment. Feel free to send me any questions, or if anything written here is incorrect or incomplete.
thanks for sharing this information. I found it very handy.