Configuring a Virtual I/O Server

Mark R. Bush

Virtualization is a hot topic these days, fueled in part by VMware, Xen, and other impressive technologies. Many commercial Unix vendors now offer server virtualization solutions of some sort. Leading the way by drawing from its mainframe virtualization heritage is IBM with System p5 servers. IBM's virtualization technology is grouped into an orderable feature known as Advanced Power Virtualization (APV). APV enables micro-partitioning (granular to 1/10 of a CPU), Virtual I/O Server (VIOS), Shared Ethernet Adapters (SEA), and partition load manager (PLM). Utilizing the vSCSI and SEA functionality inherent in VIOS, you can create many reliable and robust logical partitions (LPARs) on a single server.

The definition of virtualization is quite varied. One might argue that RAID is a simple form of virtualization -- taking many disks and presenting them to the OS as one LUN (virtually one disk). One definition that I have adopted is taking some physical "real" compute resource and representing it in such a way that allows it to be used by multiple other systems. A simple example using IBM System p5 servers might be dividing up a server with four physical CPUs into eight LPARs, each with its own portion of the CPU's cycles and its own unique and separate version of the complete AIX or Linux environment.

Virtual I/O Server

VIOS software comes with your server if purchased with APV feature. It is an AIX 5.3 mksysb (system image) customized as a specialized appliance that serves up I/O to client LPARs. It has limited functionality and is not to be used for any function other than I/O. The VIOS is loaded into the LPAR of your choice. This LPAR must have the Virtual I/O server OS type set in the LPAR profile. There are a few ways to load the VIOS image. Using NIM, from the HMC, or simply from a DVD resource available in the VIOS LPAR. My preference is the latter.

After you have the physical DVD resource assigned to the VIOS LPAR, activate the LPAR and open a virtual terminal to that LPAR. When prompted, select the DVD resource as the boot device, and the VIOS will then begin loading. When finished, you will be presented with a typical AIX login screen. The only difference here is that you'll need to login with the default user padmin and the default password of padmin. Before you can do anything on the VIOS (like change the password), however, you'll need to accept the license agreement by typing license -accept. You are now free to use any of the appliance's commands to configure the VIOS to server vSCSI and configure Shared Ethernet networks via SEA. Type help to see a list of available commands.

Two VIOS Are Better than One

I recommend using two VIOS per physical machine for redundancy and concurrent maintenance. Simply create a second VIOS LPAR. You'll need a duplicate set of physical resources (e.g., NICs, HBAs, etc.) that matches the first VIOS you created. For example, if I create a VIOS with one dual-port NIC and two HBAs, I would create the second VIOS with another dual-port NIC and two more HBAs assigned to it.

Virtual Ethernet and Shared Ethernet Adapter

Built into the hypervisor is the ability for all LPARs (AIX 5.2 and higher) to create and use virtual ethernet. This is not a function of APV and can be used without a VIOS installed. Virtual Ethernet is facilitated in hypervisor memory and acts just like a regular Ethernet adapter except that it's capable of bus speed transfer (using larger MTUs). Virtual Ethernet has only one major limitation. It can't communicate outside the physical machine. So Virtual Ethernet is good for communicating LPAR to LPAR but not good if you want that LPAR to participate on the regular network. Enter Shared Ethernet Adapter (SEA).

SEA uses a "real" physical NIC or EtherChannel to pass Virtual Ethernet traffic from the LPAR to the regular network of your choice. SEA acts as a layer2 switch and has all the 802.1q capabilities you may need. For a comprehensive discussion on that, refer to the VIOS documentation. For purposes of this article, to create a SEA that can send out your Virtual Ethernet traffic, you must first ensure that you have a physical NIC assigned to the VIOS LPAR(s). The following steps show how to create a SEA within the VIOS:

1. Ensure you have a Virtual Ethernet adapter and at least one physical NIC or ether channel device available:

lsdev -type adapter

2. Create a SEA by issuing:

mkvdev -sea ent4 -vadapter ent2 -defaultent2 -defaultid 1

where ent4 is the "real" NIC, and ent2 is the Virtual Ethernet adapter on the VIOS. Defaultid is the VLAN you want this adapter to participate in (this VLAN is the internal VLAN number not necessarily the VLAN in your regular network environment -- see documentation). The end result will be a new Ethernet adapter (usually the next number in order -- in this example, en5).

3. Assign the newly created SEA an IP address using the mktcpip command. (e.g., mktcpip -inetaddr 192.168.0.50 -netmask 255.255.255.0 -interface en5 -gateway 192.168.0.1 -hostname VIOS1). Now, from a client LPAR with a Virtual Ethernet participating on VLAN 1, you should be able to access your regular network as long as that Virtual Ethernet is properly configured (with proper IP addressing, gateway, netmask, etc.).

To add redundancy to this configuration, create the same configuration on your second VIOS and create an additional Virtual Ethernet on each VIOS on its own VLAN to establish a SEA failover channel in the event that one VIOS comes down or you want to do maintenance. See VIOS documentation for more information. With the network all set, it's on to Virtual SCSI.

Virtual SCSI (vSCSI)

Prior to POWER5 architecture, you needed physical disk or disk I/O adapters dedicated to each LPAR. Thus, if you wanted 6 LPARs on a p650 with access to its own disk resources, you needed enough PCI slots and/or disk bays to accommodate each of those LPARs. With APV and VIOS, you can now assign those physical resources to the VIOS, and the LPARs can access those resources through the VIOS. The vSCSI architecture uses a client/server methodology. The VIOS is the server, and all LPARs that use VIOS resources are client LPARs. As with SEA, you need to create Virtual SCSI adapters on both the VIOS and the client LPAR. The following steps will detail how to virtualize SAN disk on the VIOS and map it for client LPAR use:

Configure VIOS with a Virtual SCSI Server Adapter

1. At the HMC, select the physical server where your VIOS is located and expand Partitions.

2. Expand the LPAR to show the partition profiles associated with it.

3. Right-click the profile and select Properties.

4. Select the Virtual I/O Adapters tab.

5. Click the Create Server Adapter button.

6. Put in a server slot number (default is next in order).

7. Select "Only selected client partition can connect" and then select the client LPAR from the drop-down list. (You may need to change the client slot number later.)

Configure Client LPAR with a Virtual SCSI Client Adapter

1. At the HMC, select the managed system -> expand Partitions -> select the client LPAR's profile -> right-click to select Properties -> select Virtual I/O Adapters -> Client SCSI.

2. Click Create Client Adapter -> enter a client slot number (default is the next in order) -> enter the Server Partition and the Server Partition slot number you created on the VIOS -> click ok to finish.

3. On the VIOS LPAR profile, select the Virtual I/O Adapters tab -> create server adapter and enter the slot number you gave in Step 2 -> Click "Only select client partition can connect" -> select the client LPAR from drop-down list -> enter client LPAR slot number -> click ok.

Configure the VIOS Disk Resources

1. Open a terminal to the VIOS and login (user padmin and password padmin are the defaults).

2. Type lsdev -type disk to see the disks presented to the VIOS through "real" HBAs. (Depending on the SAN you use, it may be necessary to use the vendor's multi-pathing software, such as EMC's PowerPath or IBM's SDD.) Ensure the SAN disk you wish to present is there; if not, work with your SAN admin.

3. Run the lsdev -virtual to find the vhost adapters that are available on the VIOS. Each vhost represents a virtual SCSI server adapter.

4. Find out which slot number each vSCSI server adapter is in by running the lsmap -all command. Look in the Physloc field (e.g., U9111.520.10FXXXX-V3-C10). The number just after the -C is the slot number you selected when you created the vSCSI server adapter.

5. To present the whole LUN to the client LPAR make note of the hdisk# (hdiskpower or vpath in EMC or IBM) you wish to present and map it to the client LPAR with:

-- mkvdev -vdev hdisk# -vadapter vhost# -dev <name>

where hdisk# is the hdisk you want to map to the adapter, vhost is the vSCSI Server adapter you identified in Step 4, and name is an arbitrary but meaningful name of the device. The name is optional but useful.

6. Now when you do a lsmap -all, you should see the LUN you just mapped. If the client LPAR is activated, you'll see the client LPAR's ID number under the Client Partition ID field.

MPIO and Multiple VIOS

Again we are in a sticky situation if this one VIOS goes down. The process is identical for the second VIOS on your system. The only caveat is that your SAN administrator will have to present the same LUN to both VIOS on the system (not an uncommon request and one used frequently in clustering). Once you have the LUN presented, vSCSI server and client adapters created, and everything mapped properly, you should be able to run the lspath command on the client LPAR. You'll see two vSCSI adapters, and the LUN will now be able to go through either VIOS allowing you to do maintenance on one VIOS while the other serves up vSCSI and SEA requests.

With IBM's Virtual I/O Server technology, you could consolidate many older pSeries servers into new System p5 servers and potentially reduce the number of I/O adapter cards you need in your systems. Using Shared Ethernet Adapter, you can use virtual Ethernet and pass that traffic to the "real" network. With Virtual SCSI, you can provide a reliable way of virtualizing disk to client LPARs. Happy virtualizing!

Mark R. Bush is a Solutions Architect for Solutions-II Inc. (http://www.solutions-ii.com), a premier IBM business partner, where he designs, builds, and implements virtualization solutions for companies in the west. He has 10 years experience as an AIX, Solaris, and Linux systems administrator and is fascinated by all *nix type operating systems. In his spare time, he enjoys spending time with his family.