virtualization.info

Welcome to a new virtualization.info column. Here we will explore a largely uncharted territory: virtualization security.

Researchers have been working on the topic since some time now and vendors have been releasing security guides or best practices on a regular basis: yet, most of the real-world security challenges lying in virtualization have not been fully discussed nor integrated in the security governance of most enterprises.

When we think about virtualization security, we surely think about highly technical issues, involving CPU registries, resource locking and time-based covert channels. We will start this column by hosting a series of articles where we’ll try to show that this is not just what virtualization security is all about.

In this column we will always try to focus on differences between what we know very well, something that can be easily managed thanks to industry best practices, standards and experience, and what is lying just underneath the surface, the unknown (or at least, not well known) risks we are facing in the new virtualization world.

Indeed, we will start our first run of articles by discussing operational issues, starting from the low-hanging fruits of wrong management and lack of common sense.
We will then move into more technical matters, exploring how aggressors are likely going to attack a virtualization infrastructure.
We will also explore issues lying in hypervisors themselves, then discussing what role the software ecosystem can play when it comes to security, and where completely new threats can came from.
We’ll close this first series focusing on small details, where the devil lies: subtle differences which we have to take into account when we think about security and virtualization, small details which can make a difference between a defaced machine and a safe one. We will focus on hardening and how virtualization interacts with it: as we will see, this is not always an happy marriage.

Let’s start with the Template Management in the Virtual World (aka Old Threats Made New).

Template Management in the Virtual World (Old Threats Made New)

One of the most noticeable improvements in server management which is directly connected to virtualization is what is generally known by the name of "templates".

Even if vendors are referring to this concept with different terms (Profiles, Clones and so on) the basic idea remains the same: a skeleton virtual machine is created at some point to be re-instantiated at some future date to speed up the provisioning of new production servers or ease testing and development.
This is just a part of what we will call Secure Virtual Machine Lifecycle, something we will often cover in this column, but a very interesting one due to its common adoption in relatively young and small infrastructures as well.
Even if no direct support for templating is provided by the hypervisor, such a feature is somehow emulated by storing a "golden master" of a virtual machine which is then copied over and over again.
Similar practices are well known to experienced system administrators, and are not exclusively related to virtualized environments: disk imaging software, tools like Microsoft Sysprep or even remote image dumpers have been around since a long time and have been widely leveraged in large enterprises.

Why, then, is this templating stuff relevant to the virtualization security community?
To answer we must focus on the differences between what we had before virtualization and what we have now.

To begin with, we are currently missing security best practices in template deployment.
With disk imaging solutions a sysadmin would boot the system as soon as the content of the DVDs or external disks had been copied on the machine: according to industry best practices he would do so in an isolated network in order to, for instance, change passwords and properly configure and harden the server in a safe environment.
What about a virtualization-endowed environment? The administrator would most likely boot the machine right in the middle of a production or development virtual switch and proceed to change the passwords and configure the services afterwards, leaving the server open for a certain time-frame to a various range of attacks. This is not, by itself, a technological issue: the administrator could easily deactivate the network interface and safely configure the server.

However, as experience in the security field suggests, everything which is not either a standard procedure or a strict technological process is usually not implemented, and what would be just common sense during a standard deployment (just unplug the cable) is easily overlooked when working with a virtualization management console.

It's not just a matter of passwords and default configurations: in a world where remote, exploitable attacks are identified so often, an unpatched server can be exploited in a matter of minutes. Surely the security team in charge of checking updates and possibly performing the upgrade will take care of the server at some point, but any conscious system administrator would, at the end of a full fledged installation process, perform any and all available security updates before the server is lent to the application management team.

What is going to happen in a template-based virtual environment? Is that template, which has already undergone the installation of core, complex baseline software, going to be modified to apply security updates on a timely fashion? Or is it going to be untouched for months, each server instantiated thereafter being exposed for at least a short vulnerability window while updates are installed?
The answer as you might have guessed is the latter in most environments, and virtualization vendors have acknowledged the fact by releasing tools to directly handle templates, like VMware Update Manager (VUM) and Microsoft Offline Virtual Machine Servicing Tool. Yet, ISV support for the tools could be improved, and their adoption rate has not skyrocketed since their availability.

Another issue with templates comes with attacks to the template database. Where is the administrator going to store the golden masters?
Is the storage system going to have a production level system security, or are the templates going to be regarded as commodity tools, left in some network share and easily accessible by an attacker?
As I had the change to explore myself with a custom proof of concept tool, it's trivial for an attacker with write access to an offline golden master to implant rootkits which will be then carried over to any virtual machine based on the compromised template.

While the issues we described have much to share with attacks to any traditional template-based infrastructure they are much more relevant to the virtualization environment: awareness on the matter is rising between vendors and security-wise virtualization administrators.
Proposed solutions, however, are nowhere near comprehensive: until a secure virtual machine lifecycle  management is fully integrated in standard hypervisor management systems organizations should develop and enforce proper security policies and procedures to ensure that no threat can came from this vector.

In the next part of this series we are going to cover the security of virtualization management web consoles. Stay tuned!

Labels: Security