Automate System Configurations and Changes with cfengine

John Borwick

Humans are not as good as computers at applying changes consistently. If you manually maintain several computers, your machine configurations will drift apart over time. Aside from the problems inherent in manual maintenance, there are several benefits to automated, repeatable processes. Repeatable processes help you understand and document your machines' environments. If you can automate system changes, you can repeat your configuration process. You can develop disaster-recovery systems that rely on reinstalling the OS and applying changes, rather than recovering your OS from backups. Automating your system configurations also means less work for you over time.

Cfengine, written by Mark Burgess, is one of several tools available to automate system configurations and changes. Cfengine lets you assert certain conditions on a server, and over time the server will converge towards meeting those conditions. Depending on how you write your cfengine configuration files, the first time cfengine runs on your mail server it could install your sendmail RPM. The second time it could update /etc/mail from a CVS repository. The third time cfengine runs it could restart your mail daemon if it's not running already. At midnight, cfengine might clear out all old files from "/tmp" and run your log rotation scripts.

Cfengine works by reading its configuration files, setting "classes" for the machine on which it runs, and then processing actions in an "actionsequence" of things to do. Classes set the conditions under which an action occurs. Your actionsequence define the types of actions -- like deleting files or mounting NFS shares -- that should be taken.

Every machine in your environment can use the same configuration files. These files will be processed differently depending on which classes have been set. For example, the same configuration file could set your Solaris and Red Hat Linux machines to have identical "/etc/resolv.conf" files, but might only install "atop" on your Red Hat Linux machines.

This article will introduce you to cfengine and help get you started. For additional information on cfengine, I recommend the Cfengine Reference Guide and other documentation at http://www.cfengine.org/documentation.phtml, the cfengine wiki at http://cfwiki.org, and the cfengine-help mailing list at http://lists.gnu.org/mailman/listinfo/help-cfengine.

Relevant Directories and Files

All the files important to cfengine are stored in "/var/cfengine":

bin/       Binaries
inputs/    Configuration files
modules/   Third-party cfengine code
outputs/   Time-stamped messages and complaints
ppkeys/    Public and private cfengine keys
state/     Environmental data

Binaries in /var/cfengine/bin are typically symlinked to somewhere such as /usr/sbin for ease of use.

Configuration Files

Unless you explicitly specify a full path, cfengine assumes all configuration files are in /var/cfengine/inputs. Cfengine's two most important configuration files are /var/cfengine/inputs/update.conf and /var/cfengine/inputs/cfagent.conf. When cfengine runs, it will evaluate both update.conf and cfagent.conf.

Update.conf is parsed first; it's designed to be your emergency configuration file. Update.conf typically sets your system time and copies the rest of your cfengine configuration files from your cfengine server. The system time has to be set for cfengine to authenticate with the cfengine server, similar to how Kerberos authentication requires synchronized clocks. The cfengine configuration files get copied to the client in update.conf so that, if you botch the rest of your configuration, update.conf can restore your configuration with a new (corrected) copy. Update.conf should get your cfengine client back to a "known state". For these reasons, your update.conf should have no dependencies and be very small.

Cfagent.conf is parsed second. It controls all the work you want done on any of your cfengine clients. Because environments usually have one set of configuration files, and machine-specific actions are controlled via classes, your configuration tends to be quite large. One of cfengine's keywords is import, which allows you to include other cfengine configuration files. Our site's cfagent.conf includes many import statements, so we can split up our configuration into manageable files.

Syntax

Each cfengine configuration file is divided into actions, conditions, and declarations:

action:
  class1::           # 'class1' is a condition
    declaration
    declaration
  class2|class3::    # 'class2|class3' is a condition
    declaration

Conditions, or "compound classes", are always denoted by double colons. Conditions can get fairly complex, but at their simplest level, a condition is just a class name. You can use the special class "any" to apply an action statement to all machines.

Conditions can be unions of classes (logical "or") by using a pipe. Hr00|Hr12 means execute at midnight or noon, for example. Conditions can be intersections (logical "and") by using a period. Hr00.disk_space_needed means execute when it's midnight and the class disk_space_needed is defined. You can negate classes with a bang. !Hr00 means do this unless it's midnight. You can also use parentheses with conditions. (!Hr00.(disk_space_needed|Tuesday)) means run the action statements if it's not midnight and either disk space is needed or it's Tuesday.

Classes

Cfengine defines many classes (also known as groups) when it runs. You can define your own classes in configuration files, too. On one Red Hat Enterprise Linux 3 machine, many classes are predefined. I will present them here by logical group:

10_1_1 10_1_1_1 ipv4_10 ipv4_10_1 ipv4_10_1_1 ipv4_10_1_1_1
192_168_1 192_168_1_1 ipv4_192 ipv4_192_168 ipv4_192_168_1 ipv4_192_168_1_1
net_iface_eth0 net_iface_eth1 net_iface_lo
host1 host1_example_com

August Day20 Hr21 Hr21_Q1 Min10 Min10_15 Q1 Yr2005

any cfengine_2 cfengine_2_1 cfengine_2_1_10 compiled_on_linux_gnu

32_bit i686 linux linux_2_4_21_4_ELsmp linux_i686 linux_i686_2_4_21_4_ELsmp
linux_i686_2_4_21_4_ELsmp__1_SMP_Fri_Oct_3_17_52_56_EDT_2003
redhat redhat_as redhat_as_3

cfenvd also helps cfengine define more classes, such as entropy_www_in_low, based on measurements about the current environment. I have omitted these classes from the list above because they make the issue of learning about classes more complicated. You can see all the classes defined on your machine by running cfagent -pv and looking for the line that begins with "Defined Classes".

To define your own classes, you can use the classes action. The classes action will define a new class if one of the classes you list is defined. For example, if you're going to use cfengine to install software packages on different types of machines, the most useful predefined class is the machine's hostname:

classes:
  any::
    mx_server = ( host1 host3 host11 )
    mail_server = ( mx_server host12 )
    cfengine_client = ( any )

Using conditions, you could rewrite the above section:

classes:
  host1|host3|host11::
    mx_server = ( any )
  mx_server|host12::
    mail_server = ( any )
  any::
    cfengine_client = ( any )

The Control Action

Of all the actions in your cfengine configuration files, control is the one action that will always get run. The control block controls what cfengine should do and how it should do it. In the control block, there are several variables you can set. For example, you can set how often cfexecd should run with the special schedule control variable:

control:
  any::
    schedule = ( Min00_05 )

The most important control variable is the actionsequence, which defines what actions you want to run. With the exception of the special actions control, import, alerts, and classes, your actions will not run unless they are listed in the actionsequence.

Actionsequence

Section 4.9.2 of the Cfengine Reference Guide lists all the following valid actions for your actionsequence:

mountall mountinfo checktimezone netconfig resolve unmount packages
shellcommands editfiles addmounts directories links mailcheck mountall
required tidy disable files copy processes module:name

links -- This action creates soft links. It can create simple links, or it can create a "forest" of links. This declaration means "link everything in /usr/local/Sophos/bin into /usr/local/bin":

links:
  sophos_server::
    /usr/local/bin +> /usr/local/Sophos/bin

shellcommands -- The full path of shell commands to run. Some commands can do strange things with standard input or output that can make cfengine hang, which is why some cfengine shellcommands are written like:

"/path/to/binary < /dev/null > /dev/null 2>&1"

editfiles:
  redhat::
    { /etc/sysconfig/rhn/up2date
      LocateLineMatching '^forceInstall=[1-9]'
      ReplaceLineWith 'forceInstall=0'
      DefineClasses 'up2date_forceinstall' }

directories -- This simple action creates directories.

tidy -- Given a pattern of files to find and a minimum age, tidy deletes files.

disable -- Renames files. The renamed files get the suffix .cfdisabled appended to them. Disable can be used to ensure you have no /etc/hosts.equiv files lying around on your servers.

files -- Checks file permissions and can optionally maintain a checksum database.

copy -- Copies files from one directory to another, or from your cfengine server to cfengine clients. cfservd must be running on your cfengine server, the server must allow the client to talk to it, your machines must have the same clock time, and the client and server must have each others' public keys in /var/cfengine/ppkeys.

processes -- Checks to see whether processes are running, and can define classes under certain conditions.

Example

Now that you understand classes, the control block, actions, and actionsequence, let's look at an example. See Listing 1.

Deployment

Cfengine can be deployed in two ways. You can put a schedule line in your control section and run cfexecd like any other daemon, or you can add a cron entry to call cfexecd. In our environment, we run cfexecd as a daemon, but we could have just as easily added the following to cron:

0 * * * * /var/cfengine/bin/cfexecd -F

Typically, people who deploy cfengine have a version-controlled repository where they check in configuration updates. The cfengine server checks out the most recent configurations. Cfengine clients then copy the configurations from the cfengine server. This version-controlled server-client configuration is a bit of a pain; we set it up in our environment with help from Luke Kanies's article, "Distributed Cfengine", available at:

http://www.onlamp.com/pub/a/onlamp/2004/05/13/distributed_cfengine.html

Potential Problems

Besides the difficulty required to set up a cfengine server/client environment, you may run into a few other problems with cfengine. As hinted at in the example configuration file, cfengine parses all your imported files before it executes them. As cfengine reads your configuration files, it creates a list of potential class dependencies. If you import files "A" and then "B", and file "B" defines a class, file "A" will never see that class defined. You must keep track of the order in which files are imported.

Also, your actionsequence order is really important. You will eventually get the temptation to re-order your actionsequence because your "links" action really needs to use your "shellcommands" action. My advice is to pick an order and stick with it. Cfengine runs multiple passes through your actionsequence to resolve class dependencies and, if you really need to, you can use a more advanced technique and run the same action several times with different defined classes. See the Cfengine Reference Guide, section 4.9.2, for details.

Cfengine defines its default classes when it first runs. If you add an IP address to a machine within a cfengine configuration file, for example, cfengine's default classes will not include that IP address until the next time you run cfengine.

cfagent Options and Testing

cfagent, the command-line tool to run cfengine, has 46 options. Here are the 6 that I think are most useful:

-K -- Don't wait on any cfengine locks before running; run now.

-v -- Be verbose.

-q -- Don't wait the default "splay time"; start running immediately. By default, cfengine waits a random amount of "splay time" before running to keep cfengine clients from overwhelming the cfengine server or other communal resources.

-n -- Perform a dry run. Don't actually change the system.

-D<class> -- Define whatever class you specify when running the configuration file. For example, when our site is ready to fully initialize a machine, we run cfagent -Kvq -DInit. That means ignore locks, tell me everything, run right now, and run with the "Init" class (which for us means "it's OK to do things that might break the system").

-f<file> -- Process the specified file rather than update.conf followed by cfagent.conf. When testing cfengine statements, I typically create a file called /var/cfengine/inputs/test.conf, with a tiny complete configuration file that contains whatever I need to test. For example, the file might say:

control: actionsequence = ( tidy )
tidy: /tmp recurse=inf age=0

The other time I use -f is when I just want to run update.conf. To update our cfengine server's copy of /cfserver, the server only has to process update.conf. I can run cfagent -Kq -f update.conf on our server to make new versions of our configurations available for our cfengine clients and save it from running through the whole configuration in cfagent.conf.

Conclusion

Our environment defines classes like mysql_server and lsi_client to install the appropriate binaries and utilities, and to copy our configuration files. Cfengine is a timesaving maintenance tool once you have expended the effort to configure it initially. Cfengine's advantage and complexity comes from being able to automate virtually all parts of server configuration.

John Borwick is a SAGE Level III systems administrator at Wake Forest University, with a background in Computer Science and English. His goal is to make systems administrators' lives easier by optimizing common processes.