Conserver: An Update on the Open Source Console Management System

Bryan Stansell

How would you like the flexibility to troubleshoot system problems from anywhere? How would you like the ability to have others watch what you are doing, and even help? And how would you like to have everything logged, so you could easily review past events? These are just some of the things conserver can do for you, and in this article, I'll review these features and provide some updates on the development of conserver.

Conserver is a daemon that runs on a Unix-like system to facilitate remote management of devices. The only prerequisite to taking advantage of conserver is the ability to connect to the "console" of the device you wish to manage. This "connection" can be via a direct serial connection, a TCP socket, Unix domain socket, or any command conserver can invoke. A TCP socket is traditionally used to connect to a terminal server, whereas Unix domain sockets were added to support VMware consoles. And, random commands allow you to do anything -- from connecting to proprietary systems to integrating into an IPMI environment.

Once connected, conserver can log everything displayed on each console. This allows administrators, for example, to review any error messages sent to a console immediately before a system crash, which can be of enormous value when debugging problems remotely. Conserver also manages users accessing the consoles, entering commands, sending break signals, etc. It even supports multiple users accessing the same console simultaneously.

Building Conserver

Conserver has been run successfully in a variety of Unix-like environments. This includes the Cygwin environment for Windows. Any POSIX-based environment that the GNU configure utility supports should work. All the common vendors work just fine.

Some conserver-specific configuration options are PAM (Pluggable Authentication Module) support, tcp_wrappers (for further access restrictions), OpenSSL (to encrypt and authenticate client-server communication), and Unix domain sockets (to have all client-server communication contained on a single host). There are also various options for tuning the server. The documentation can provide additional details.

The Configuration File

The configuration file format is very powerful, especially for such a simple concept as a console server. It is formatted into blocks of information, with an "include" facility to allow for inheritance of information and elimination of redundant data. These blocks of information are console definitions (of course), sets of named defaults, runtime settings, access lists, groups of users, and break strings (similar to macros). All blocks of information are of the form:

type name {
  key value;
  key value;
  ...
}

To give you an idea of what this looks like, here is a very simple, completely valid, configuration file (which could even be on a single line, if you like):

console simple {
  master localhost;
  type exec;
  rw *;
}

This creates a console named simple that is managed by the local conserver host. The console type is exec, which means it will run a command (in this case the default of /bin/sh -i) and all valid users have read-write access. Console items take a variety of key-value pairs, the most important of which tell conserver which host should manage the console (see the Distributed Conserver Hosts section), how to connect to the console, where to log the console data, and who can access the console.

The console option initcmd specifies which command should be attached to the console when it first comes "up" (i.e., when conserver successfully connects to the console). This allows you to authenticate to a terminal server or do any other activity required to get to the point where you are fully connected to the console. There is also a timer (initspintimer) and counter (initspinmax) to catch rapid successive failures of initcmd and stop conserver from thrashing.

Aside from protecting itself, conserver can also protect you by doing port calculations internally. This allows you to do things like specify the physical port number in the console definition and let conserver calculate the TCP port number or even local serial port name (using the devicesubst feature, which is described below). That can be quite handy when you are on the phone with a data center technician and want to say which physical port a machine is plugged into. Conserver uses the formula portbase + portinc * port.

Console items, when fully specified, end up having a lot of common information. This is where sets of defaults come in handy. They allow you to take all the common information and combine it into logical groups, which you can then use in a console definition (or other groups). You can group defaults in default items, which take the same key-value pairs as console items.

All default items are given a name. The default named * is special and automatically applied to all further console definitions. Consoles (or other default items) can explicitly reference a default by using the include option. Here's an example of a named default (ts-ssh) being included in the console definition for foo:

default ts-ssh {
  type exec; portbase 2000; portinc 1;
  exec /usr/local/bin/ssh -p P -l tsuser H;
  execsubst H=hs,P=Pd;
}
console foo { include ts-ssh; host ts1.conserver.com; port 4; }

This may look a bit complicated, because I haven't explained the execsubst, devicesubst, and initsubst options. They perform substitutions on the exec (command-based consoles), device (local serial port), and initcmd options. The substitution allows you to replace characters in the option with new strings, formatted to your preference, based on the value of the host option, console name, port option, and calculated port value.

The previous example substitutes all H characters with the host option (which is set in the console definition), and all P characters with decimal representation of the calculated port value. Thus, you can define an abstract connection in a named default and only put console-specific information in the console definition. If the default was named * instead of ts-ssh, the include ts-ssh; could have been removed since the * default is automatically included in the console definition.

You can also change the runtime behavior of conserver via config items. These match the command-line options you can give conserver. For example, you can set file locations, change SSL settings (if you compiled in SSL support), disable the client redirection feature, and set the default access type.

With managed consoles, you must also think about access restrictions. Access to a console is governed by two levels of restrictions: server-specific and console-specific. A client must first be allowed to connect to the server, and then it must be allowed to connect to the particular console.

The first server-specific access restriction is governed by tcp_wrappers (if built into the code). The next access check is against the lists built with access configuration items. If no access list matches, the default access type (defined with the defaultaccess option in the config item) is used.

To build an access list, the name of the item should match the conserver host it applies to or * for all hosts. There are three types of access checks, as shown in this access list for conserver01.conserver.com:

access conserver01.conserver.com {
  trusted 127.0.0.1;
  allowed 192.168.0.0/16, 172.0.0.0/8;
  denied 192.0.0.0/8;
}

The first match is used, so connections from 127.0.0.1 are allowed in without a password; connections from 192.168.0.0/16 and 172.0.0.0/8 are allowed if a password check succeeds; and connections from the rest of 192.0.0.0/8 are denied. You can specify hostnames, specific IP addresses, or netblocks in the access lists.

Once past the server-level access list, the client must then be given access to the console via the rw and ro console options. As I showed previously, the rw option specifies who is allowed access to a console in read-write mode. The ro option specifies who is granted read-only access. Both options take a list of users, which can be user names, conserver group names, or system group names (/etc/group, etc).

Conserver group names are created with the group item. You give it a list of users, just like the rw and ro items, and can use them wherever a list of users is needed. Here's an example:

group admin {
  users bryan, todd, @sysadmin;
}

This defines a group named admin, which consists of the users bryan, todd, and everyone in the system group sysadmin. To use the group, you just specify it like rw admin;.

If a client is to have access, and a password check is necessary, conserver looks for a conserver.passwd file. That file should have user names and encrypted passwords of the form username:passwd. If the user exists and the password matches, access is granted. Two special keywords are allowed in the file. For username, you can use "*any*", to match all users. And for passwd you can use "*passwd*", which tells conserver to look up the password via PAM (if compiled in) or the standard system library. This way you can hard-code passwords for some users, allow others to use their system password, restrict access by locking an account, or do a variety of other combinations. If the conserver.passwd file doesn't exist, conserver will just query PAM or the system library.

One final aspect of the configuration file is its ability to be split into parts. This allows you to edit pieces individually, generate a portion based on external data, or organize things however you like. You simply specify a file that you'd like to include via the #include feature:

#include /etc/conserver-globals.cf
console one { include ts3.conserver.com; port 4; }

The conserver-globals.cf file should define the ts3.conserver.com default for the above to work. The #include directive must not be preceded by whitespace, otherwise it will be treated as a comment.

These are just some of the ways in which you can define defaults, consoles, users, and access lists. There are more examples of basic to complex configurations in the conserver.cf/samples directory of the distribution.

Distributed Conserver Hosts

Conserver also supports the idea of a distributed conserver configuration. The idea is that you would have multiple hosts running conserver, sharing a single configuration file. Each server would then redirect the client requests to the appropriate conserver host. That is what the master console option states -- which host should manage the console. One way to set this up would be to override the automatic default (*) as you define consoles:

default global { ... } # set generic defaults here

default * { include global; master conserver1.conserver.com; }

console foo { include ts1.conserver.com; port 2; }
console bar { include ts2.conserver.com; port 3; }

default * { include global; master conserver2.conserver.com; }

console zip { include ts1.conserver.com; port 4; }
console zap { include ts2.conserver.com; port 5; }

The first two consoles are managed by the conserver host conserver1, while the second two are managed by conserver2. Both conserver hosts know where all consoles are managed and can properly redirect clients to the proper conserver host.

There really is no difficulty in setting up a distributed configuration. You could even take independent conserver.cf files from multiple conserver hosts, concatenate them, and distribute them (assuming your master values were unique hostnames and not localhost). But using things like named defaults and sharing the information between console definitions would, of course, make things much cleaner.

The Conserver Client

The conserver client can be used in two ways: to do global operations across all consoles and to connect to a specific console. All global operations return you to your shell, whereas connecting to a console virtually links your terminal to the console.

Most of the global operations are querying functions. You can ask who is attached to each console, if the consoles are up, how the consoles are connected (host and port numbers, local devices, etc.), and even get a formatted data dump of the state of the consoles for scripting purposes.

Aside from the querying functions, you can also do "bulk" operations. You can send a broadcast message to all users, send a message to specific users, disconnect users, and tell the conserver processes to shut down. The destructive functions require administrator rights (which are set with the admin option in config items), but the rest are available to all valid users.

When connecting directly to a console, the client does nothing but send all key sequences to the server and display all characters the server returns. There is an escape sequence, processed on the server, that allows you to do other things, like bring the console up or down, send messages to other users on the console, take over read-write mode (only one user is allowed read-write access at a time), send break sequences, etc. To give you an idea of all the things you can do, here's the help screen you get when connected in read-write mode:

$ console simple
[Enter '^Ec?' for help]
[help]
 .    disconnect                      ;    move to another console
 a    attach read/write               b    send broadcast message
 c    toggle flow control             d    down a console
 e    change escape sequence          f    force attach read/write
 g    group info                      i    information dump
 L    toggle logging on/off           l?   break sequence list
 l0   send break per config file      l1-9 send specific break sequence
 m    display the message of the day  o    (re)open the tty and log file
 p    playback the last 60 lines      P    set number of playback lines
 r    replay the last 20 lines        R    set number of replay lines
 s    spy mode (read only)            u    show host status
 v    show version info               w    who is on this console
 x    show console baud info          z    suspend the connection
 |    attach local command            ?    print this message
 <cr> ignore/abort command            ^R   replay the last line
 \ooo send character by octal code

You can set client defaults in a system-wide console.cf file or a user's .consolerc file. The format of the file is the same as server's configuration file (key-value pairs). You can set things like the default escape sequence, conserver hostname and port, number of relay and playback lines, SSL options, etc. The client configuration file is also the way to define custom strings that are displayed when you attach and detach from a console. If you're using a terminal with a status line or title bar, this is an ideal way to have it updated. Here's an example for xterm and GNU screen, which also changes the escape sequence and playback length:

config * {
  escape ^Ee;
  playback 200;
}
terminal xterm {
  attach "^[]2;console: U@C^G";
  attachsubst U=us,C=cs;
  detach "^[]2;^G";
}
terminal screen {
  attach "\033_console: U@C\033\134\033kconsole: U@C\033\134";
  attachsubst U=us,C=cs;
  detach "\033_\033\134\033k\033\134";
}

As you can see, using the client is straightforward and provides a wide array of capabilities and information, which is exactly the way it should be.

Conclusion

By no means was this a complete description of everything conserver can do. I hope, however, that I've provided a good overview of why conserver is a robust, mature console management system. The latest version of conserver can be found at:

http://www.conserver.com/

or one of its mirrors. The mailing lists and archives are available to help you make conserver a successful part of your environment.

Bryan Stansell has been part of the system administration community in the San Francisco bay area for more than 10 years. Although he enjoys hacking on open source software, nothing compares to the joy his wife and new daughter bring him. Bryan can be contacted at: bryan@conserver.com.