Article Figure 1 Listing 1 Listing 2 Listing 3
Listing 4 may2006.tar

Worldwide Access to Your Serial Consoles

John Fox and Mark Uris

Imagine being able to reboot a server from the system console from Europe or at an airport during a layover. Imagine not having to physically be present at your datacenter to perform tasks that require console access but, instead, doing them in the comfort of your own home or office. Now imagine doing all this with relatively low-cost commodity components running your favorite Unix-variant operating system along with an open source software package.

Like many computing centers, we recently experienced some growth and needed a way to reduce our space/power/heat footprint in the datacenter. Our use of physical character console and keyboards attached directly to servers has given way to racks filled with high-density headless servers, KVM switches, network cables, and power strips.

We connected the serial consoles of these servers into a single server running open source console management software that can be accessed remotely using one-time passwords (OTP). The savings in space, power, and heat alone are impressive, but we have found that the big benefit is in having remote console access capability from home and office. We currently manage around 50 Sun servers and are increasing this number steadily. This article gives an actual example of our experience with building a centralized console management solution, starting with the high-level design of the overall system down to the actual implementation details.

Why Conserver?

The conserver software package was written by Bryan Stansell to allow multiple systems administrators to access system serial consoles remotely. Stansell wrote an excellent article on conserver in the Console Servers supplement of the September 2005 issue of Sys Admin magazine: 

http://www.samag.com/documents/s=9842/sam0514b/0514b.htm
Stansell also gives an overview of conserver in a FAQ on his Web site: 

http://www.conserver.com/FAQ
In his article, Stansell brought up the main functionality of conserver that attracted us as a way to get remote console access to our servers. Our previous experience with remote system access was with Sun servers, such as the E250 and E450, that were equipped with the Remote System Console (RSC) card, an Intel co-processor running a stripped-down command set. This required additional network connections and never really caught on with the Sun user community. Conserver presented a whole new approach to remote access and also provided:

    1. Global, secure access from anywhere with an Internet connection. Sys admins can perform patch updates, go into single-user mode, or monitor console messages from anywhere with the appropriate level of encryption, such as VPN.

    2. Logging of all console output to a central place for troubleshooting and later review.

    3. Multiple users can access the console with only one user having read/write access.

Serial Console vs. KVM

For environments with PC/Intel platforms, a KVM (Keyboard-Video-Mouse) switch is needed to access your machine, rather than a serial console. For our initial configuration of a serial console system, we used systems that required only serial access, namely Sun servers, although HP and IBM machines fall in this category as well. The use of KVM switches is beyond the scope of this article.

Software and Hardware Needed

We used the following to build a serial console server:

    1. A machine with two or more network interfaces and a reasonable amount of memory and disk.

    2. A serial console switch; there are a number of good switches on the market.

    3. The conserver software package, which is available at: 

    http://www.conserver.com

The conserver software has been compiled on several different operating systems, including Sun Solaris, Linux, FreeBSD, and AIX. Supported operating systems are listed on the conserver Web site.

Our configuration uses the console server as a "gateway" server. On one side of the gateway, systems administrators log in using one-time passwords over the public Internet. The server is heavily secured, and all unnecessary services and ports are turned off. Behind the gateway server is a private network (see Figure 1) for access to each serial console switch (we have three of them).

Install and Configure the Server

Our current production console server is the second system we have used. The first one was an old Sun Ultra 1 we had lying around. We ran Solaris 8 on this server and made sure it was heavily secured and patched on a regular basis. This server worked well, but we needed to upgrade to accommodate more serial consoles. We chose two Sun Cobalt servers (one is a hot-standby backup server). This allows us to bring down the production server and still have access to all of our serial consoles, which helps us sleep better at night. These two machines came configured with Sun Linux (a Red Hat variant) and mirrored system disks (RAID 1) as insurance against data loss. These systems were tightly secured and access could only be had by using one-time passwords. The only active service is ssh.

Network Configuration

The servers are configured with two network interfaces. The first interface is plumbed on our internal network and accessible only from an internal host or via a VPN connection from outside networks. The second network interface is for the Ethernet connection to each console switch. This second network is a private address space, accessible only to users that log in to the console server itself. Listing 1 provides an example /etc/hosts file that shows both the public and private network configuration.

Install and Configure the Console Switch

A console switch allows you to aggregate your serial console connections into one place. The switch has an Ethernet connection that allows you to telnet to the switch and select the console port of interest. Most serial console switches will accommodate between 8 and 32 serial consoles. We used the Cyclades TS2000 console switch with 32 ports. The Cyclades unit comes factory-configured so that attached Sun servers will not "break" to the >ok prompt in the event of a switch power loss, a well-known issue with Sun servers. Prices for commercial console switches vary, but as of this writing $55 per port seems to be a good rule of thumb.

In our configuration, we use multiple console switches. We bought an inexpensive unmanaged Netgear 8-port switch (http://www.netgear.com), so that all three of our console switches can share the second interface of our console server.

Install the Conserver Software

To begin, download the tar file from http://www.conserver.com. Unpack and read the installation instructions, found at: 

http://www.conserver.com/INSTALL
In our experience, we had no difficulties in the first compile with the native c compiler on the Sun Solaris system, or with the gcc compiler on the Sun Linux systems.

There are two configuration files that had to be customized for our particular environment. They are conserver.cf and conserver.passwd. Password entries in conserver.passwd can be null if you want to allow a user access to any console. Because the systems administrators needed access to all the different servers, we set all the accounts to null. This can be seen in Listing 2.

In some environments, it may be necessary to use multiple console switches to accommodate a large number of servers. These switches can be configured using the conserver.cf file. In our environment, we had to adjust this file to accommodate our switches. Listing 3 shows an example of a conserver.cf file for multiple console switches.

Getting Onto the System Console

All that's left now is to access the console. Be sure that each user is defined in /usr/local/etc/conserver.passwd, even if there is a null password. Also be sure to have the following defined in /etc/services: 

console         782/tcp         conserver       # console server
Now give it a try: 

# console <hostname>

server1 console login:
At this point you are attached to console. To verify this, try the "who" command: 

server1 console login: [who server1]
 admin@conserver.mydomain.com   * attach   0:00 Mon Feb 13 06:50:05 2006
You should be able to log in to the server console at this point. There is a potential security risk, however. Remember to log out of your console session when you are finished; otherwise, another user can simply attach to the console and assume your session. It is no different from leaving yourself logged into a real, physical console, and then walking away without logging out.

The commands available to do the various console functions are extensive (see Listing 4). Once the basic sequences are mastered, however, it is fairly straightforward to use conserver to conduct system console activities. A number of features, like the "force attach read/write", allow you to force off a user who is tying up the console access, for example. As with all sys admin activities, practice makes perfect. The more experience you have with the commands, the more intuitive they become. We strongly recommend playing with the commands for a while so that, when real intervention is required, their use is automatic.

Although the conserver package allows only one sys admin to attach to the console in "read/write" mode, up to 16 other administrators are allowed to attach to view activity being conducted. This activity can be viewed just like a regular console session -- that is, other administrators who are attached can see the same activity as the administrator attached in "read/write" mode, but they cannot write.

System Logging

Console output from all attached servers is logged to /var/consoles/<hostname> by default. These files benefit you in a number of ways. First, if a system is compromised or hacked into, all the system console logs are available for review. Second, central logging is convenient, because logs are aggregated in one place, similar to a central loghost setup for logging events via syslogd. Third, the sys admins seem to be very much aware when they are using the conserver as a server console and tend not to leave open sessions around. This adds up to better systems administration practices.

Troubleshooting

There are occasions when getting console access fails for some reason. You can verify console connectivity with the "show host status" command (Listing 4). This will give a listing of all the systems that are online. In most cases, the loss of connectivity is caused by a switch losing power or connectivity with a server, in which case the switch must be rebooted. We have had occasional issues with the Sun Enterprise 250/450 series, which required repeated attempts to establish the connection to the serial console. These issues are rare and easily fixed in most cases. On occasion, we've had to simply restart the conserver process on the console server.

Conclusion

Our experience using conserver has been extremely positive. It has allowed us to conduct after-hours sys admin activities from home, such as installing patch updates where console access is needed to work in single-user mode. The convenience of being able to view what sys admin activities has prevented two admins from being logged in at the same time via ssh and also facilitates easier troubleshooting of server issues.

Conserver is scalable and allows us to add more console switches and servers with ease. We will continue to use the conserver package and appreciate the efforts of Bryan Stansell for his continued support and updates to the software. We invite other systems administrators to discover the benefits of remote console access and centralized console management.

Resources

Conserver -- http://www.conserver.com

Cyclades -- http://www.cyclades.com

Netgear -- http://www.netgear.com

Zonker's Greater Scroll of Console Knowledge --http://www.conserver.com/consoles/gsock.html

John Fox is a systems administrator specializing in Sun/Solaris and has worked in a variety of environments including ISP, Telco, and academic research. He is currently employed at the National Center for Atmospheric Research in Boulder, Colorado. His interests include infrastructure management, high availability, systems security, and server performance tuning. John can be reached at: johnrfox@gmail.com.

Mark Uris has been doing systems administration within the Scientific Computing Division of the National Center for Atmospheric Research for more than 17 years. He's had a close working relationship with Sun and has beta-tested a number of Sun hardware and software products. Of late, he has been involved in Teragrid technology for NCAR, as well as expanding his area of expertise to Linux, OpenBSD, and other open source systems. He can be reached at: uris@ucar.edu.