Securing your VMware ESXi Hypervisors

Securing your VMware ESXi Hypervisors

Background

In recent months virtualized hypervisor environments such as VMware ESXi have increasingly been targeted by adversaries with the sole objective of financial gain, leveraging ransomware as a tool of choice. In a typical ransomware attack, adversaries aim to compromise the corporate domain and distribute the ransomware via their preferred method to all accessible assets within the domain. Although virtualized infrastructure, such as backend hypervisors, is a prime target due to its critical role in running vital business services, it is often left untouched since the malware strain cannot execute its functions on unsupported operating systems. However, more recently developed ransomware strains have emerged, such as ‘Nevada’[1], with specific support for ESXi and the encryption of virtual machines.

The shift in the threat ecosystem is caused by may factors, including but not limited to:

• Lack of endpoint protection solutions (AntiVirus/EDR) offering support for hypervisor platform architecture, resulting in a lack of visibility and detection opportunities.
• Unpatched vulnerabilities, many of which are historical and are not considered a priority within the patch management process, despite having significant CVSS ratings.
• Insufficient security configurations, enabling ease of access to ESXi shell or no preventative measures.
• Lack of security monitoring and alerting to logs not being forwarded to a centralized security information and event management ().
• Lack of enforcement of multi-factor authentication (MFA) when authenticating user accounts.

The aforementioned items offer significant opportunities, increasing the appetite amongst adversaries to target. 

Notable attacks in the wild

Over the last few years, two significant advisories were released by VMware, both of which pertained to the service location protocol otherwise known as OpenSLP, which is a network service installed by default on versions of ESXi hosts. The vulnerabilities were both assigned significant common vulnerability scoring system (CVSS) ratings and should be mitigated in a timely manner:

• On October 20^th, 2020, VMware issued an advisory VMSA-2020-0023.3[2], following the identification of a remote code execution (RCE) vulnerability CVE-2020-3992 (CVSS 9.8 Critical).
• On February 23^rd, 2021, VMware issued an advisory VMSA-2021-0002[3], following the identification of a heap-overflow vulnerability CVE-2021-21974 (CVSS 8.8 High).

OpenSLP is used to enable ESXi applications to discover and resolve network related information and other service detection within a local area network (LAN). The service is typically exposed via port 427.

In recent weeks adversaries operating new ransomware strains such as ‘ESXiArgs’ have been conducting widespread scanning of port 427 to exploit ESXi hosts with vulnerable OpenSLP network services exposed. If successful, this results in arbitrary code execution with root permissions on the ESXi host, which then facilities the next steps of post-exploitation discovery and ultimately data encrypted for impact.

Vulnerabilities are not always at fault for provisioning initial access to ESXi hosts; insufficient security controls and a lack of network segmentation often allow for lateral movement to ESXi infrastructure. Adversaries performing network discovery will often identify ESXi servers that currently have active secure shell (SSH) sessions connected to a System Administrators’s host. This offers the opportunity to hijack the session which has not been terminated successfully or remains idle.

Malicious payload types

Malware specifically compiled for encrypting ESXi hypervisors identified in the wild is typically formed of a series of shell (.sh) scripts, python (.py) scripts or/and Executable and Linkable Format () binaries. Once a malicious payload is dropped onto the ESXi host, the payload must be assigned execute permissions prior to being run, which can be achieved via chmod +x ./payload.sh .

In order for ransomware to disrupt virtualized assets, the payloads ultimately need to render virtual machine disk images (VMDKs) and configuration files inaccessible by encrypting them using an asymmetric key. This can easily be achieved by traversing the VM files within the datastores and running the encryption routine.

VMware vSphere encompasses a command line interface framework known as ‘ESXCLI’. This enables administrators to execute a catalog of commands to perform tasks, such as listing virtual machines located within datastores or changing system configurations. This is beneficial to an adversary because it can be used to perform system discovery and identify virtual machines available to target by executing a single command such as esxcli storage filesystem list.

While very uncommon, it is also possible to compile a custom vSphere Installation Bundle (VIB) package for nefarious purposes, which could be installed using ESXCLI onto an ESXi host. VIBs are mostly only created by VMware and trusted partners for use.

A VIB package is essentially an archive file containing three files:

1. Descriptor (XML) – The descriptor file contains the VIB package details, such as the contents of the archive and requirement information.
2. VIB (VGZ or TGZ) – This is an archive file containing the payload which will be installed on the ESXi host.
3. Signature (PKCS7) – The signature file contains the electronic signature for trust verification and additional details for integrity purposes. If unsigned, it can contain an empty signature file.

In a hypothetical situation, an adversary could compile a VIB package containing a malicious payload, and install the unsigned package using ESXCLI command esxcli software vib install -d <payload>, which once executed, runs malicious code to encrypt data on the ESXi host.

Security Recommendations

VMware advisories should be followed to ensure all security patches are applied to affected systems to mitigate any identified vulnerabilities. Additional steps should also be considered to address the vulnerabilities, but also to reduce the risk of VMware ESXi compromises:

• VMware ESXi infrastructure should be segregated from your local area network (LAN) to prevent lateral movement.
• Utilize local accounts with a different password and MFA enforced, separate from standard corporate domain admin accounts for ESXi management.
• Although remote SSH and ESXi shell services are disabled by default, they are often left enabled after carrying out system administration tasks. These services should be strictly controlled by implementing additional restrictions:
• Disable SSH and ESXi shell services when not in use.
• Only permit SSH access from specific authorized IP addresses.
• Restrict direct console user interface (DCUI) and SSH idle time, instead of unlimited which to prevent session hijacking.
• Implement lockdown mode to restrict access. Exception lists can be configured to allow specific authorized users access if required.
• Disable unnecessary services frequently attributed to many vulnerabilities:
• Disable OpenSLP network service, if running. OpenSLP is enabled by default in older ESXi versions and is typically not required. It has also been attributed to several vulnerabilities exploited by adversaries during recent campaigns.
• Ensure the ESXi host is situated behind a firewall and restrict unnecessary ports, including port 427 (OpenSLP).
• Carry out auditing and configuration reviews of vSphere Installation Bundles (VIB):
• VIB package acceptance levels should be reviewed, and any ‘CommunitySupported’ acceptance level bundles interrogated to determine legitimacy.
• Enable ‘execInstalledOnly’ enforcement. This will prevent the execution of any VMkernel binaries, which were not part of an authorized signed VIB.
• Forward log files to a SIEM to enable monitoring, detection, and investigation of ESXi activities:
• Monitor ingress communication to port 427 (OpenSLP) for abnormal exploitation attempts.
• Monitor ESXi host access for any abnormal user account authentications.
• Monitor logs for unusual activities, i.e. unauthorized VIB installation, and abnormal command.

Conclusion

Organisations must ensure the issues discussed in this report are addressed in order to reduce the exposure of hypervisors to an unauthorized adversary. Patch management programs must be implemented to ensure vulnerabilities are patched in a timely manner and security controls must be enforced to restrict the ability of misusing ESXi hosts for nefarious activities.

[1] https://www.resecurity.com/blog/article/nevada-ransomware-waiting-for-the-next-dark-web-jackpot

[2] https://www.vmware.com/security/advisories/VMSA-2020-0023.html

[3] https://www.vmware.com/security/advisories/VMSA-2021-0002.html