Navigating the System Virtualization Maze -- Part 1

Virtualization holds an undeniable appeal at many computing facilities, and for very good reasons. However, navigating the path from desire to your own "virtual reality" is no simple feat. The trail is filled with obstacles, complexities, and difficult decisions. There are myriad technologies, both hardware and software, and each has its own pros, cons, features, and limitations. Sometimes the path costs money, sometimes it costs time and effort, and sometimes both.

This month in the Solaris Companion, I provide a map to the landscape and the paths to system virtualization. While there are certainly other virtualization technologies (such as networking and storage), server virtualization is the most heated and most complex. Also, because operating systems can run on top of other operating systems in our virtual world, I go beyond Solaris where appropriate.

In part 1 of this column, I will discuss virtualization in general -- the options available for system virtualization on Unix systems, the benefits of virtualization and the data needed to determine the best virtualization solution for your set of circumstances. In part 2, next month, we will find our way out of the virtualization maze by turning the data into a virtualization plan and delving into the darker side of virtualization -- the complexities, limits, and tradeoffs.

Brief Virtualization Overview

Succinctly, virtualization enables a facility to pretend to be another facility. In the storage realm, one storage array could look like several arrays to ease administration and increase security. In networking, one set of cables can carry multiple virtual networks that act as if they are the only traffic on that wire.

For systems virtualization, products and product marketing have given us several variations on the theme of "one system pretending to be many systems". These can be segregated into several categories, as follows.

• Hardware partitioning -- This method involves the strict division of a system into fixed collections of CPUs. Those CPUs get their own operating system installed on them and run independently of the other partitioned CPUs on the system. Note that hardware partitioning has been available to Sun customers for many years, in the form of hardware domains on the enterprise servers. These servers also include an advanced form of hardware partitioning that can move CPUs between partitions -- Sun calls it "dynamic reconfiguration". Dynamic reconfiguration still partitions the system into separate groups of CPUs, and the resource movement is on large time scales (hours or days), so it deserves a separate category from the other forms of virtualization discussed below. Hardware partitioning virtualization is also newly available in the T1 chip (as found in the T1000 and T2000 serves. The T1 chip sports a "logical domain" facility as of the Solaris 10 11/06 release. The T1-based systems can now create up to 32 LDoms, each of which can have its own Solaris 10 installed and running. It could be argued that this physical partitioning of a system is not really virtualization, but it does involve one system acting like many so it is included here. See Figure 1A.
• Normal virtualization (also called operating system virtualization) -- This setup has hardware or software, the "hypervisor", acting like the bare metal of the system, allowing for example multiple operating systems to be installed on the same system. The virtual machines see only the virtual bare metal, and therefore do not realize they are virtualized. The hypervisor then manages the resources to share them amongst the virtual machines. Sometimes this resource management is basic (e.g., virtual machine A gets four CPUs and machine B gets two CPUs) and sometimes more advanced. Note that there is complexity to this solution. For example, many aspects of the bare metal virtualization involve emulating hardware facilities, such as page tables and interrupt vectors. VMware Workstation and ESX are examples of this type of virtualization. See Figure 1B.
• Paravirtualization -- This is a cooperative solution in which the virtualization technology exposes an API to the operating systems running on top of it. That API allows the virtualized OS to ask the paravirtualizing hypervisor to perform tasks for it (e.g., change a page table entry). In this case the complexity is reduced and, potentially, performance can be gained. However, the virtualized operating system must be modified to call the hypervisor's API. Therefore, the operating systems being virtualized must volunteer for that duty, and non-volunteers (i.e., Windows) are not invited to play. Xen and VirtualIron are examples of this type of virtualization. See Figure 1C.
• Hardware-assisted virtualization -- This setup improves both normal and paravirtualization by giving the hypervisor help in emulating the base hardware. With hardware assist, such as the Intel VT and AMD-V (as found in the Sun Opteron "M2" servers), there is less emulation and interception by hypervisors. For example, Xen could not run Windows in a virtual machine before Intel VT and AMD-V came along.
• Containers -- Containers, as found in Solaris 10 and beyond, and BSD "jails" belong in a separate category from the other technologies. They draw the virtualization line not between the operating system and the hardware but rather between the applications and the operating system. In this manner, applications can run in virtual containers on top of one operating system. This form of virtualization is less flexible than the standard models in that there is one kernel running and all applications run within that one kernel (and kernel patch set). But before condemning them as too restrictive, consider the Solaris project "BrandZ" (http://www.opensolaris.org/os/community/brandz/), which will allow other operating systems to run (virtually, of course) with a Solaris Container.

Containers are much lighter weight than other system virtualization forms and are easier to manage. For example, Sun supports 8,192 containers within a single system, and patches install in the "global" container as well as all other containers on the system. This is in stark contrast to other technologies. See Figure 1D. Containers were previously discussed in Sys Admin Magazine here:

http://www.samag.com/documents/s=9494/sam0502b/0502b.htm

Virtualization can be useful on both workstations and servers. Most companies concentrate on the server side, but QA groups and individuals enjoy the freedom that workstation virtualization provides. As an example of the astonishing power of workstation virtualization, consider my recent teaching experience. At the USENIX LISA 2006 conference in Washington D.C., my class on Solaris 10 administration had 110 students attending. Because this class was in workshop form, I needed a hands-on environment for the students to try various examples and exercises.

As a radical experiment (note to self: it's best to avoid experiments when teaching classes), I started with my Mac Book Pro (core 2 duo), added the Parallels virtualization software, and installed Solaris 10 (actually the open source "Nevada" build 54). Inside Solaris I built 60 containers. The students then telnet'ed into those containers to do their work. Consider that 110 users logged into 60 Solaris virtual environments, all inside the Parallels virtual environment running on MacOS X! Virtualization clearly works and is clearly useful in a wide variety of circumstances.

Benefits of Virtualization

The benefits of virtualization, when configured and deployed properly, are astounding. Many sites have successfully consolidated many systems (8 and 12 are commonplace) down to one system. Not only does this free up the old hardware, but it also decreases rack space, power, and cooling use. It also saves on maintenance expenses and administration time.

There are some situations that are perfect for virtualization, and many others that are very good virtualization targets.

• Systems under low load and running single applications are the best example. Underutilized systems can be freed up for other uses, and applications and operating systems can be flexibly deployed and managed. The benefits of installing single applications on dedicated servers, specifically independence of applications from each other and from conflicting operating system configurations, are still gained.
• Many sites use virtualization to improve disaster recovery. Most virtualization technologies allow an entire virtual-system image to reside in a single file. Replication of the server is then a simple matter of replicating that image to the DR site. Of course, virtualization must be in place at the DR site as well. Also, virtualization software typically presents a generic hardware environment to the virtualized operating systems. The remote site can have differing hardware from the production site, but the virtual systems and applications will run without issue because they see that same generic hardware at both sites.

Virtualization can further ease DR implementation by allowing applications at the DR site to be consolidated onto fewer systems than the source site. Other optimization of resource use can occur at the DR site, such as manually juggling running applications to make the best use of limited hardware. In production, nightly batch runs may be on separate servers instead of daily-use interactive applications. In DR, these applications may be combined on to the same systems with manual fine tuning of resource use, all to decrease the cost of systems at the DR site.

• Virtualization allows rapid restoration of a virtual machine from a saved state (i.e., previous virtual machine disk images). In case of disk crash or data loss, the backed-up file image can be restored and service restored much more quickly than when restoring a system from tape.
• Another benefit of the separation of the virtual operating system from the hardware is that support windows can be increased. For example, an application might need to be run on an older operating system because of supportability issues. In this case, the legacy application/operating system combination can be run on modern hardware, with the virtualization software supporting the new hardware and devices that the legacy operating system could not.
• Some sites are using virtualization to increase the availability of applications at their sites and to optimize resource use and increase application performance. Most virtualization solutions implement fine-grained control of CPU and memory use. In this way, runaway processes and memory leaks affect only the individual virtualized system, not the entire physical system.

The VMWare ESX "VMotion" and Xen "live migration" features allow a virtual machine to move between servers without interruption of access to the application. These features can be used to move applications from an overloaded server to one with more available resources, seamlessly. They could also be used when a system needs maintenance. Rather than shutting down the system and its applications, the sites seamlessly move the applications to another server, perform the maintenance, and move the applications back with no change in service. Scheduled system maintenance can be "virtually" eliminated in this way. Of course, storage and network administration may still require downtime to the applications, with little overall net gain in application availability. As usual, the facility must be considered as a whole when trying to increase availability.

• Advanced features of some virtualization can increase security. Consider a site taking a "snapshot" of a virtual machine when it is installed, and then taking periodic snapshots throughout the life of the virtual machine. If a problem occurs within that VM (security or otherwise, actually), previous snapshots can be used to debug the cause of the problem and undo any changes that caused the problem.

Feeding the Decision

Given all the virtualization options, how do you go about choosing "the best" virtualization solution for your environment and circumstances?

The first step in such a project is to gather all of the data needed to make an informed decision. The necessary information includes:

• What problem are you trying to solve (resource optimization, disaster recovery, high availability, QA flexibility, and so on)?
• Do you need differing operating systems on the same system?
• Do you need differing operating system versions on the same system?
• How do you apply patches today? Will that same method apply once systems are virtualized?
• Likewise, how are backups currently implemented?
• How many applications are involved?
• What are the applications and their versions?
• How many systems are in use that could be virtualized?
• What are the resource use levels on each of these systems (CPU, memory, disk, network)?
• What management of resource use is needed in the target environment?
• What is the security stance of each of these systems?
• What is the architecture of the network that the systems connect to?
• What are the uptime requirements of each application and each system?
• Do you need to be able to easily convert a physical system into a virtual machine ("P to V")?
• Do you need to be able to easily convert a virtual machine into a physical system image ("V to P")?

The Next Steps

In part 2 of this column, I'll start from these data points and analyze the information to determine the best virtualization solution for your environment. Virtualization is not a primrose path, and we will also look at the darker aspects of the technologies and their deployment. By the end of part 2, you should have a complete picture of the virtualization options, the choices to be made, and all of the gory details that can get you into or out of the virtualization maze.

Peter Baer Galvin (http://www.petergalvin.info) is the Chief Technologist for Corporate Technologies (www.cptech.com), a premier systems integrator and VAR. Before that, Peter was the systems manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines, and previously wrote Pete's Wicked World, the security column, and Pete's Super Systems, the systems management column for Unix Insider (http://www.unixinsider.com). Peter is coauthor of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials and given talks on security and systems administration worldwide.