La nested virtualization It has gone from being a lab rarity to becoming a key component for setting up complex environments for testing, training, development, and even certain scenarios in the public cloud. Being able to run virtual machines within other virtual machines opens up a huge range of possibilities, from setting up a mini data center on your laptop to launching a complete training campus in Azure or testing new versions of hypervisors on AWS without breaking the bank.
In this article we're going to explain exactly what nested virtualization is and how it's implemented in VMware, Hyper-V (Azure) and AWS EC2This article covers its requirements, main real-world use cases, performance implications, networking, licensing, and best practices. The goal is that, by the end of this article, you'll have a comprehensive and practical understanding of when it's relevant to use it, how to set it up, and what limitations you might encounter.
What is nested virtualization?
When we talk about nested virtualization, we are referring to the ability to run a hypervisor as a virtual machine within another hypervisor running on physical hardware, and in turn, be able to create VMs within that guest hypervisor. That is, a VM within another VM, with several layers of virtualization on top of each other.
In this context, several terms are used that should be clear to avoid confusion: the hypervisor that runs directly on the physical server is the host hypervisor, while the hypervisor that we install inside a virtual machine is usually called guest hypervisorVMs that run directly on the physical host are called external guests, and those that run inside that guest hypervisor are called internal guests. internal guests or nested virtual machines.
From a technical point of view, the guest hypervisor believes it is communicating with real physical hardware, but in reality it is working against a virtualized hardware presented by the host hypervisor. If you have sufficient resources, you can even chain more layers (layer 4, layer 5, etc.), although performance and management become considerably more complex, and in practice, it rarely goes beyond two or three layers.
Hardware-assisted virtualization and basic requirements
For nested virtualization to work reasonably well, it is essential to have hardware assisted virtualization (Intel VT-x, AMD-V, or equivalent). These extensions allow the hypervisor to handle context transitions and instruction-sensitive processes much more efficiently than with older techniques such as binary translation or paravirtualization.
In practice, this means you must ensure that the physical server's CPU supports these extensions and that they are enabled in the BIOS/UEFIIn addition, the host hypervisor must have the ability to expose these virtualization extensions to the guest operating system so that the hypervisor running as a VM can in turn create 64-bit virtual machines and take advantage of hardware acceleration.
Another important requirement in VMware environments is the level of virtual hardware of the VM which will host the guest hypervisor. To enable nested virtualization, at least hardware version 9 of the virtual machine is required. From then on, more recent versions of Workstation, Player, Fusion, and ESXi include explicit support for virtualizing CPU extensions within VMs.
Hypervisors and compatibilities
Nested virtualization is available in both Type 1 (bare metal) and Type 1 hypervisors. type 2 hypervisorsIn the VMware ecosystem, ESXi is the classic type 1 hypervisor, while Workstation, Player, and Fusion are type 2 hypervisors that are installed on top of a host operating system.
These products can run another ESXi as a VM, or even third-party hypervisors such as Microsoft Hyper-V or VirtualBoxprovided the host hypervisor allows internal hardware virtualization (HV). Broadly speaking, the technically supported combinations would be:
- ESXi as a VM in VMware Workstation, Player or Fusion.
- ESXi as a VM in ESXi (environment fully nested within the VMware stack itself).
- ESXi as a VM on third-party hypervisors (for example, on Hyper-V).
- Non-VMware hypervisors such as VMs on ESXi, Workstation, Player or Fusion, as long as they are able to take advantage of hardware virtualization.
In Hyper-V, nested virtualization is primarily used in contexts such as Azure Lab Serviceswhere it's possible to enable Hyper-V within a lab VM to create nested virtual machines. And on AWS, the arrival of official support for nested virtualization to EC2 standard (not just bare metal instances) has been a major leap forward for many testing and development environments.
Official support and restrictions in VMware
Although nested virtualization works very well in VMware in practice, it's important to consider the aspect of official supportVMware does not consider nested VMs as a supported configuration for production environments, which means that if you open a support case and the issue is related to a nested environment, it is very likely that they will not provide support.
The major exception in the VMware ecosystem is the use of vSAN Witness ApplianceA nested ESXi virtual machine acts as a witness in vSAN Stretched Clusters architectures. In this case, VMware explicitly recognizes and documents this usage, as it forms part of the product's official architecture.
There is another relevant scenario: the Microsoft virtualization-based security (VBS) In Windows VMs. Since vSphere 6.7, VMware has supported enabling VBS within Windows virtual machines, which in practice means running Hyper-V and certain virtualization isolation features within a VM. This scenario is well-documented and is covered by support if you meet the version and configuration requirements.
Licensing in nested environments
When installing ESXi, vCenter, or other vSphere components as virtual machines to set up a nested environment, it's important to keep in mind that licensing is exactly the same than if they were installed on physical servers. Each instance of ESXi, even if virtual, requires its own license, or alternatively, the free edition or trial versions can be used for lab environments.
This applies whether you deploy nested ESXi on top of VMware or run them as VMs on third-party hypervisors. A common way to optimize CPU licenses in these labs is to play with the number of cores per socket assigned to the ESXi VM (cpuid.coresPerSocket parameter), provided that the license conditions allow it and you respect the limits of the contracted edition.
Real-world use cases of nested virtualization in VMware
The great appeal of nested virtualization is that it allows you to mount highly complex environments on a single physical host without the need to deploy server racks, storage arrays, or dedicated switches. Some of the most common uses in VMware are particularly clear.
Training and self-learning laboratories
Perhaps the most typical case is that of the training laboratoriesYou can deploy multiple ESXi virtual machines, a vCenter Server, and a VM for shared storage (virtual NAS, vSAN, or similar) on a single physical host and almost completely replicate a production vSphere environment: clusters, HA, DRS, Storage DRS, vMotion, etc.
This approach is perfect for an administrator to set up a home laboratory on your home PC or server, or for internal training at a company. The fact that everything runs nested greatly simplifies testing: if a student breaks a virtual ESXi host or the nested VMs, simply restoring the external ESXi VM from a backup will get everything back to normal.
Development of solutions for vSphere
Teams that develop applications or integrations specifically for VMware (plugins, backup tools, automation, orchestration, etc.) typically need repeatable environments where raise specific versions of ESXi, vCenter, and different network and storage configurations.
Thanks to nested virtualization, you can deploy multiple test environments very quickly, with different combinations of versions, clusters, and policies, without needing a large fleet of physical servers or expensive storage arrays. You start the environment, run the test, and then destroy or clone it as needed, all on the same host hypervisor.
Tests, updates and demos
Another very powerful use is that of the testing new versions of hypervisors or architectural changes. Before upgrading ESXi on physical production hosts or introducing a new storage solution, many administrators set up the nested environment, simulate the upgrade, and validate that everything works as expected.
Similarly, sales and pre-sales teams often use nested vSphere environments to do let's live For customers: they can set up several ESXi hosts, a vCenter, a virtual storage server, and some VMs, all within a single physical lab machine or even a powerful laptop, and can demonstrate features such as vMotion, HA, DRS, or SRM without relying on a full technical room.
Mixed environments and public clouds
Nested virtualization is also leveraged to allow managed service providers to host VMware on public clouds or on heterogeneous platforms. A practical example is deploying a virtual ESXi on a public cloud host and hosting a client's VMs there, while maintaining their tools and procedures.
Furthermore, the possibility of doing nested environment backup In several ways (backing up the VM containing the guest hypervisor or the nested VMs individually) it offers a lot of data protection flexibility in these hybrid scenarios.
Performance of nested virtual machines
In terms of performance, we need to be realistic: each additional layer of virtualization introduces overhead. In VMware, for every running VM there is one process on the ESXi host which consumes RAM and CPU. If within one of those VMs you are in turn running an ESXi with several nested VMs, you are actually accumulating several layers of processes, context switches, and scheduling on the same physical cores.
In practice, nested VMs are usually slower than regular VMs. The magnitude of the impact depends on factors such as the performance of the physical processor, the amount of available memory, the type of storage, and, above all, the number of nesting levels and concurrent VMs. For light lab workloads, it's perfectly manageable, but for demanding production It's not usually the best option.
VMware tools specifically for nested ESXi
When you install ESXi inside a virtual machine, it's also important to have VMware Tools for that virtualized ESXi. It is the set of drivers and services that allows the host hypervisor to communicate better with the guest VM and improve its management.
In the case of nested ESXi, VMware Tools provides features such as displaying the IP address and hostname of the virtual ESXi from the vSphere client, and the ability to orderly shutdown or restart The nested ESXi from the management console and the execution of scripts when its power state changes, plus support for operations within the VM through the Guest Operations API.
In older versions of ESXi 5.x, these tools had to be installed manually using a VIB package. Starting with vSphere 6.0, the necessary tools are included. integrated into ESXiTherefore, when you install ESXi on a VM in the standard way, VMware Tools appears directly as present and running, greatly simplifying daily administration.
Network and security policies in VMware nested environments
The network plane is one of the most challenging aspects of setting up a nested ESXi environment. By default, ESXi virtual switches were not designed with this in mind. hypervisors within VMsTherefore, certain security policies need to be adjusted so that nested VMs can communicate correctly with the outside world.
On a physical ESXi host that will host nested hypervisors, the vSwitch (or the corresponding port group) must be modified by enabling the promiscuous modeThis allows MAC address changes and accepts frames with forged source MAC addresses (Forged Transmits). If these parameters are left at the default value (Reject), the virtual switch discards traffic from nested VMs because the effective MAC address does not match that of the guest ESXi virtual NIC.
Promiscuous behavior and its impact
Promiscuous mode is a security policy that causes a network interface to receive all L2 plots that pass through the vSwitch, not just those directed to its own MAC address. It is typically used for network monitoring, traffic analysis, and, in our case, so that the nested hypervisor can see the traffic of its internal VMs.
Enabling promiscuous mode on a vSwitch or port group makes it behave more like a hub: all traffic is forwarded to that interface. This makes it easier for the ESXi virtual machine to learn and manage the MAC addresses of nested VMs, but in return reduces network performanceThis is especially noticeable if the loads are very traffic-intensive.
MAC changes and counterfeit transmissions
By default, a standard vSwitch in ESXi does not allow a VM Change your MAC address Regarding the configuration in the VMX file, this is a security measure against ARP poisoning attacks and identity spoofing. Similarly, outgoing frames whose source MAC address does not match the VM's actual MAC address are blocked, considering them forged transmissions.
In a nested scenario, internal VMs typically have different MAC addresses than the guest ESXi host. If these policies are not set to Accept, the virtual switch on the physical host will consider frames leaving the virtual ESXi host with different MAC addresses to be suspicious and will forward them. discard, breaking the connectivity of nested VMs with the rest of the network.
Starting with vSphere 6.7, Distributed Virtual Switches (VDS) incorporate options for MAC address learning through the macManagementPolicy option, which reduces the need to use promiscuous mode and provides behavior more similar to that of a traditional physical switch in nested environments.
Key steps to set up a nested ESXi environment
In practice, the typical deployment of a nested ESXi environment on a physical ESXi host involves a series of relatively standard steps, although with some critical nuances to ensure everything works properly.
First, a virtual machine that will act as ESXi hostIn the wizard, you choose an appropriate hardware compatibility level (for example, ESXi 6.5 or later), select “VMware ESXi” as the guest operating system, and in the hardware section, allocate at least 2 vCPUs, 8 GB of RAM or more, and a sufficient virtual disk (minimum 32 GB for ESXi 7.x, although it is preferable to provide more).
The key point is to select the option to Exposing hardware-assisted virtualization to the guest operating system. If this is not done, during the ESXi installation on the VM a warning will appear indicating that hardware virtualization is not supported or is not enabled on the CPU, and it will not be possible to start nested VMs within that virtual ESXi.
Once the VM is created, the ESXi installation ISO image is mounted from a datastore, and the hypervisor is installed on that virtual machine as if it were a physical server: the disk is formatted, the management network is configured with an IP address and hostname, and access is verified. VMware Host Client via browser.
Nested ESXi storage and datastores
With ESXi 7.x, the partitioning scheme has changed compared to previous versions. On small disks (below 128 GB), a VMFSL partition is created for system data (coredump, tools, scratch, etc.), and there may not be enough space to create a VMFS datastore additional space to host nested VMs.
Therefore, it's very common to add one or more additional virtual disks to the guest ESXi VM. This involves first shutting down the machine, editing its settings, and creating new disks with thin provisioning (for example, 30 GB, 50 GB, or more, depending on the needs). Then, the nested ESXi host client detects this new disk and creates a VMFS datastore (usually VMFS 6) to store the internal VMs.
Mounting ISOs and creating nested VMs
To install operating systems on nested VMs within the ESXi guest, there are two main approaches you can choose from when use ISO images of installation:
On one hand, you can upload the ISOs directly to the nested ESXi datastore and select them from there when creating the internal VMs. This approach is simple but consumes storage space within the nested environment itself. On the other hand, you can mount the ISO to the guest ESXi VM's CD/DVD drive from a datastore on the physical host and, within the virtual ESXi, configure the nested VM's CD/DVD drive as the "Host Device," taking advantage of passthrough.
This second method is usually preferable because it allows centralizing the ISOs in a single datastore on the physical host, saves space on the guest ESXi datastore, and provides a lightweight better performance media access. Once the ISO is mounted, the process of creating the nested VM is similar to that of any other VM: the name, type of guest operating system (Windows, Linux, etc.), resources (CPU, RAM, disk) are defined and the installation is started.
After installing the operating system on the nested VM, it is crucial to add VMware Tools within that VM to optimize network performance, video, and shutdown and restart operations. Depending on the ESXi distribution used, the ISO images of the tools may be included or need to be downloaded from the VMware portal.
Cloning and replication of nested ESXi hosts
Once a nested ESXi host has been configured with the desired version, network, datastore, and some internal VMs, the normal next step is to want to clone that VM to have several identical virtual ESXi hosts, for example to form a lab cluster with vCenter.
Before starting to clone, it's advisable to make some adjustments within the original guest ESXi host: enable the VMkernel to follow the virtual hardware's MAC address with the appropriate parameter in ESXCLI, and remove the system UUID from the /etc/vmware/esx.conf file so that, when the clone starts, a new unique identifierAfter cloning, when powering on each virtual ESXi, it will be necessary to change its hostname, IPs, and any other settings that need to be different.
Nested virtualization in Azure Lab Services (Hyper-V)
Azure Lab Services offers a very convenient way to deliver complete labs to students or internal users using virtual lab machines. Here, nested virtualization is supported by Hyper-V as a hypervisor within the template VM. The idea is that from this template VM, Hyper-V features are enabled, the necessary nested VMs are created, and when the lab is published, each user receives their own copy of that template with everything already set up.
In this scenario, nested virtualization is only available in Windows-based lab VMsAlthough both Windows and Linux guests can run within Hyper-V, if the template operating system is Windows Server, it's usually necessary to configure a NAT network within the lab VM so that the nested VMs can communicate with each other and access the internet.
When setting up these labs, several practical aspects must be taken into account: if user accounts without administrator privileges are created, it is essential to add them to the group “Hyper-V Administrators"so that they can start and stop VMs. In addition, the VHDX files of the nested VMs must be in paths accessible to that user, and to save disk space, it is recommended to use the dynamic VHDX format instead of large flat disks."
Regarding computing power, the VM sizes designed for nested virtualization in Azure (for example, Standard_D4s_v4 and Standard_D8s_v4) are based on 3rd generation Intel Xeon Platinum processors. Since the underlying processor can vary within the same family when stopping and starting VMs in Azure, it is advisable to enable the appropriate mode. processor compatibility in nested Hyper-V VMs to minimize migration problems between different physical hosts.
Another critical point is configuring the Hyper-V VMs to shut down gracefully when the lab VM shuts down, typically using the cmdlet Set-VM to define the automatic shutdown action and prevent data corruption. And, as always, you need to carefully plan memory and the number of vCPUs per nested VM to ensure acceptable overall performance.
There are also important limitations: not all Azure VM sizes support nested virtualization, nested VMs do not have direct access to Azure resources such as VNet DNS, and only certain VMs are supported. Hyper-V as a virtualization technologyThis excludes other solutions that require hardware extensions such as KVM or VMware within that layer.
Nested virtualization in AWS EC2
Until recently, if you wanted nested virtualization on AWS you were practically forced to go to bare metal instanceswith direct access to the hardware but at a much higher cost and with less flexibility. The announcement of official support for nested virtualization in standard EC2 virtual instances has completely changed the landscape.
The update, reflected for example in recent versions of the AWS SDK for Go, enables the ability to run guest hypervisors and nested VMs within non-bare metal EC2 instances, provided they belong to compatible families (typically instances with Intel VT-x or AMD-V CPUs from the M, C, R series, etc.). This opens the door to startups and development teams Set up complex labs, advanced CI/CD platforms, or VM orchestration environments without going over budget.
In this context, the most frequent use cases include testing pipelines that replicate complete production architectures (different operating systems, network layers, firewalls, etc.), technical training platforms that give each student a mini data center within an EC2 or the development of cloud-native tools that need to manage VMs as part of their functionality (e.g., backup, security or automation solutions).
From an economic standpoint, nested virtualization support in standard EC2 can lead to significant cost reductions compared to bare metal, on the order of 60-70% in many lab scenarios. Furthermore, it offers advantages in elasticityOn-demand instances can be combined with Spot, scaled up or down according to testing needs, and environments can be provisioned in seconds compared to the usual bare metal times.
However, it's also important to keep in mind that each virtualization layer adds overhead. Nested VMs in EC2 tend to suffer a performance penalty of between 10-20% compared to an equivalent non-nested VM, so their natural place is in development, testing, proof-of-concept, and lab environments, while for critical production environments, it's advisable to carefully evaluate whether this performance loss is worthwhile.
For the ecosystem of startups in Latin America And in other cost-sensitive cloud markets, this possibility is especially relevant: cybersecurity companies can set up multi-layered malware labs in the cloud without paying for bare metal, DevOps teams can replicate production topologies in staging environments at a controlled cost, and SaaS providers can prepare complex demos for customers with real architectures without deploying physical hardware.
Finally, integration with the AWS SDK for Go and other SDKs allows for the automation of managing these nested environments, including instance compatibility verification, enabling virtualization capabilities, and orchestrating the VM lifecycle within EC2 instances through pipelines. Infrastructure as Code.
Nested virtualization has established itself as a highly versatile tool for lab-based testing and training on complex infrastructures without requiring significant hardware investments. From training scenarios with nested ESXi and Hyper-V labs in Azure to advanced testing platforms in AWS EC2, the pattern is consistent: intelligent use of this technology allows for increased flexibility, reduced costs, and an accelerated testing cycle, provided its performance, support, and licensing limitations are respected, and networks, security, and resources are well-planned.
