Dr. Dobb's Journal August 2000
Automated builds refer to a series of commands that can be executed to compile and link source code into its final deliverable format. Additionally, once deliverables are available, they can be packaged with an installation program.
So why is this such a big deal? Well, if your product is made up of several modules (each built separately), you have to integrate all these builds. To do so, you traverse the directory of each module, build the source, and establish intermediate and output directories that are common repositories. However, GUI-based development environments (such as Visual Studio IDE) encourage interactive rather than batch builds. But the advantages of creating a single, unified build process that can be run in batch mode -- automated and unattended -- can be leveraged in many ways. In this article, I'll discuss some of those advantages, whet your appetite with some implementation ideas, and suggest some features your build can incorporate.
A build server is a machine dedicated to the singular purpose of running automated builds. This machine has to be set up with all the tools required to build and package your source code. Once established, it should require an order of Congress to change the environment on your build server. Although this may sound draconian, it ensures that your automated builds are performed correctly and predictably.
It is best to assign a single person as the "build captain" who ensures that the automated build is executed correctly and regularly and with due process. Your project likely won't have an automated build to start with, especially if code is being written from scratch. However, you should deploy one as soon as possible. A good time to put an automated build in place is as soon as you add the first files to version control.
Automated builds provide many benefits, the first and foremost of which is ready-to-go source code. Code that is always ready to build saves time. Imagine a testing team preparing several machines to start a testing cycle. If they can't build the modules for testing, time is wasted for both the development team (scrambling to fix problems) and the testing team (twiddling their thumbs).
The next benefit is a reproducible build. A reproducible build successfully creates output files that are identical to those created by a previous iteration of the build. To ensure that one file is identical to another, you can compare them bit by bit via a file-comparison utility. You can use the same build process to build output files on two identical machines and still end up with differences in your output files. The files may even function in the same way regardless of the machine they were built on. However, differences in even a byte between the same file should set off alarm bells.
In theory, a single byte differential could cause code to crash in certain situations. Such differences are usually the result of subtle differences in the build environment between different machines. An incremental version difference between the compilers, a particular registry setting made on one machine but not on another, a difference in the include paths, and the like, can all cause different builds to pop out of identical machines.
Some project leaders try to solve this problem by assigning the build of a particular module to a certain team member. This supposedly ensures that the modules produced by team members are reproducible since they come from the same machine each time. But this is a risk you're better off not taking. Team members fall ill, go on vacation, or decide to quit and take a job in Hawaii. Once in a while, they might install a productivity application, say, NukeEmToPulp 1.0, which can change an environment setting or two and forever alter the build environment without anyone else knowing.
Also, consider a customer who comes to you with a bug in version 1.12. Meanwhile, your source code is now at version 2.0. It's not reasonable to expect customers to upgrade to 2.0 just so you can work on a bug. Nope, you are going to have to work with version 1.12. If you've followed the rules of source-code version control (see my article "The Version Control Process," DDJ, May 1999), you will have established a version-control label for version 1.12. Thus, you can extract all the files for version 1.12, build them, and reproduce your customer's bug. At this point, you want to guarantee that the files you build from the checkpoint are the same you shipped to the customer. A reproducible build facilitates reproducing the bug.
Another benefit is the unattended build. At some point in your project -- especially around the beta stage, when your project enters an incremental bug fixing and testing pattern -- you want to build source code repeatedly and often. An unattended build scheduled to run on the build server facilitates this. You can run the unattended build overnight and have the latest version of code ready for testing on a shared drive by morning. By giving the testing team access to this shared drive, you automate the process of handing over the latest source code for testing. In effect, the latest source code to be tested is always available on the shared drive of your build server. Better yet, you could bundle your modules with the installation program for your modules and create a directory for installation. This process also has the desirable side effect of forcing you to think about installation, which is often treated as a poor cousin during development.
In this way, you encourage the testing team to test the correct modules. This eliminates the unproductive, but common, scenario in which a test engineer, working on a particular version and unaware of updates, reports a problem that has already been fixed. However, when the latest versions of modules are available on a build server, the test engineer can come in, chug on a cup of coffee, and double-click the installation program before beginning a fresh round of testing. Not only are the most recent changes tested, but the installation program is also put through its paces every day.
Automated builds are also beneficial to developers. This feature is particularly valuable for large projects that take a long time to build. My article "The Version Control Process" discussed the need for all developers to work with the latest source code every day. This practice lets you identify integration problems almost as soon as they occur, thus keeping the project on track. If files are being changed constantly, especially a file like IncludeMeEverywhere.h that is used by virtually every module, each developer spends a substantial amount of time building modules that they don't necessarily want to test. Worse yet, all developers spend time on this task every time source code is updated.
By providing fresh builds everyday on a build server, you give developers the option of downloading the output and intermediate files directly onto their hard disk along with the latest source from version control; thus, saving the time required to build these modules. An incremental build can then be performed, which will build only the files that have been modified by that particular developer.
This productivity benefit also allows developers to pull down the retail versions of the modules and mix them with a debug build or a module that they are debugging. If there is a situation that warrants debugging an associated module, the debug build for that module can simply be downloaded from the build server and copied to its rightful place on the developer's hard disk.
Size tracking is another benefit provided by automated builds. Executable files and libraries grow in size at the rate at which code is added. Source code additions are directly related to new features or bug fixes. However, a new compiler or linker switch can also cause a significant change in the size of an output file. A function call that pulls in calls from several other libraries can also result in a significant size increase in the final output. This is usually inadvertent. Developers often don't pay careful attention to how much code they are pulling into their module when creating a function call that resides in an external library. By running automated builds daily, you are in the position to incrementally track the changes in size of output files. This lets you correct problems such as those I just described. You can also write a small utility that saves the size of your output files and compares them between builds. You can set it up to warn you if the size of a module changes beyond a given percentage. Alternately, you can chart the change in size for each module regularly and use the information when conducting code reviews.
Another tangible benefit to having an automated build is the ability to get new team members started quickly. Once all the tools are installed and the source code has been pulled from version control, a single command can build all the modules from scratch. For new team members, setting up the environment and building the source successfully represents a significant milestone.
It is important to create an automated build process in the form of one or several commands that can be invoked from a single batch file. The prevalent way of creating build processes is by writing Makefiles -- scripts that are read and interpreted by the Nmake utility. Central to Makefiles is the concept of a target and its dependents.
In Example 1(a), for instance, the target is checked against its dependents, Dependent1 through DependentN. If any of the dependents is newer than the target, the block of commands immediately following that line is executed. This checking and execution of commands is performed by the Nmake utility.
Example 1(a) can be extended to build the source code for an executable. In Example 1(b), the Alarm executable depends on several object modules. This dependency represents a rule for building Alarm.exe. If either of the object files is newer than Alarm.exe, the link command is executed, building a new Alarm.exe. The special variable $@ is set by Nmake to the target being processed. The variable $** is set to the list of dependents. You can use such variables to simplify your Makefile scripts. In my sample Makefile, I use these variables to create generic command lines for common tasks.
The object files are built if any of the corresponding source files become outdated. This rule is captured in a generic way by the .c.obj construct, which instructs Nmake to build any object file if it can find a .c file of the same name that is newer.
Nmake sets internal variables with values that can be retrieved by the $() operator. Nmake also sets some predefined variables to default values. For example, CC is set to the command cl.exe. The variable LINK is set to link.exe. You can override these definitions by setting the variables yourself, and you can reduce platform dependencies in your Makefile by using such variables. The variables LFLAGS and CFLAGS, see Example 1(c), let you reuse a set of linker and compiler flags in various places without having to repeat them throughout your Makefile.
In short, you can use Makefiles to build executables, libraries, and even perform all kinds of odd jobs -- including copying and deleting files.
To create an automated build, I'll use a sample called "AlarmClock" -- a module that provides online alarm services for hotel guests. The automated build creates the executable for this application. As Figure 1 shows, the source code is organized in separate directories: Alarm holds files related to triggering the alarm, DateTime houses files that manipulate the date and time, and UserInterface contains files that interact with users. Finally, Common holds code shared between all the directories. In this example, I use it to hold some header files.
Besides being able to build the entire project, Makefile must also be extensible. By designing the Makefile implementation carefully, you can create components that can be reused by other projects.
As you can see in Figure 2, the master Makefile is made up of several smaller Makefiles. Each included Makefile addresses a particular area of functionality. Table 1 describes the functionality of the included Makefiles. I define a variable that abstracts the root directory, \Devroot. I use this directory to find every other directory used in the project. By formalizing a directory structure for your projects, you make them more manageable and maintainable. Most developers really don't want to change the directory structure or rename directories. It's unproductive and the benefits gained can be obtained in other ways. In this Makefile, I let you specify an alternate name for the root directory only. If you define an environment variable called "PATH_DEVROOT," the Makefile will use this definition and will not create a default definition. I use this technique in the Makefile whenever I want to let you override defaults.
Once the root directory has been set up, you use it to find the directory, which is home to your included Makefiles. The first Makefile I include, Paths.mk, maps the directory structure of the project to variables that can be used in other Makefiles. You include your final Makefile, Targets.mk, at the end of the master Makefile. The Nmake utility requires you to set up all definitions and declarations before you can specify rules for targets. Since Targets.mk contains only rules, it must be included after all definitions in the Makefile have been specified. You formalize this by specifying that it be included at the end of the master Makefile, by which time you will have finished defining all variables.
Listing One (Paths.mk) sets up variables that define a temporary directory for intermediate files and a release directory for output files. These variables are used in compiler and linker flags to direct output files. Directories that are repositories for intermediate and output files have separate paths based on whether the build is a debug or retail build. This separation lets users switch between debug and retail builds without having to rebuild all the components again. Paths.mk requires a special variable called "PATH_HOME" to be defined in the master Makefile. This variable defines the directory that contains the master Makefile and will be the base directory for running the automated build. If this variable is not set, Paths.mk aborts the build using the !error directive.
Listing Two (Shell.mk) sets up definitions for commands. The only interesting thing going on here is the way the VERBOSE option is supported. When building source, especially when modifying the build process itself, you might need to look at the sequence of commands being executed by the Makefile. (Normally, you'd choose to hide these commands because they create a lot of screen clutter.) By specifying the VERBOSE=1 option when calling Nmake, you can see each command being executed. When VERBOSE is not set to 1, you can hide all commands by prefixing them with the @ operator. The intermediate variable CMDECHO holds the value of the prefix. It is set to either @ or NULL, according to the value of the VERBOSE option.
Listing Three (Options.mk) specifies compiler and linker flags. The value of these flags changes depending on the value of the DEBUG option. You use the value of DEBUG to specify compiler switches for optimization and debugging. If the NOBROWSE flag is set to NULL, you set up the compiler and linker switches to build a browser database for the source.
Finally, you define variables for commands that can compile, link, and generate a browser database. These commands can be plugged into a Makefile as required to perform their respective tasks. You will see examples of this in the master Makefile, which uses the commands COMPILEME, LINKME, and BUILDBROWSERDB. If a compiler or linker supports new switches, these can be added to the included Makefile without having to change every single master Makefile that uses the compiler and linker. Vc.mk defines variables that are useful when integrating with version control.
Listing Four (Targets.mk) sets up rules for some useful targets. This includes a Clean target to delete intermediate files. A Usage target displays help on how to use the Makefile. A Setup target creates the temporary and release directories if none exist.
The Clean and Usage targets have a double semicolon (::) in front of them. This is used in a Makefile when there are multiple rules for the same target. Nmake will process both sets of rules. Thus, a Clean target can also be setup by the master Makefile to delete some files that the included Makefiles don't know about. For example, in the master Makefile, I define an additional Clean target to clean up the output files.
Listing Six is the Makefile in the \DevRoot\AlarmClock directory. You must set up some paths in the master Makefile prior to including Paths.mk because you want to be able to find the include Makefiles first. Take the opportunity to define the PATH_HOME variable, which is expected by Paths.mk.
The master Makefile specifies additions to any compiler or linker switches setup by the include Makefiles. In the sample, you add some project-specific switches to the compiler. You then define some rules for the targets that are to be built. The first rule is for a target called "All," which is invoked every time the Makefile is parsed. The default is to build AlarmClock.exe, which depends on a number of object files that have rules specifying how they need to be built. When referring to the location of the output and intermediate files, you liberally use the variables defined in Paths.mk. Developers are thus insulated from creating these directories manually for every project.
You use predefined variables COMPILEME to compile and LINKEME to link. Developers who write the Makefile are thus insulated from the internals of how to invoke the compiler and linker.
Finally, you include the Targets.mk file at the end. That's it. All you need to do is type in Nmake and you are off to the races.
Although you now have an easy-to-use build process that can run unattended, you have one more important step to perform before scheduling automated builds on a build server -- you need some way of getting all the latest source code before building it. This requires integration with your version-control software.
Listing Five is a sample included Makefile called "Vc.mk" that can be called to copy the latest version from your version-control database onto the hard disk on the build server. This Makefile defines variables that abstract any specific version-control software. The Refresh target in the master Makefile uses the defined variables to pull source code. The reason you provide a separate target is because you don't want refreshing source code to be a conscious decision made by the person who invokes the Makefile. If the target were integrated in such a way that it was invoked every time a developer wanted to build source, you would open Pandora's box.
By creating a structure for writing project- based Makefiles, you can create an automated build process for a project. Such a process provides a model for creating unattended, centralized, and repeatable builds. These features can be leveraged in many different ways to streamline a project.
DDJ
#############################################################################
# \DevRoot\Build\Include\Paths.mk
# Include Makefile for automated build process.
# Defines directory paths for stock directories used for the build.
# (c) Aspi Havewala 1998
#############################################################################
!ifndef PATH_HOME
!error Please define a variable called PATH_HOME. The directory
specified by this variable will be hold your intermediate files.
!endif
_T_PATH_TEMP = $(PATH_HOME)\Temp
_T_PATH_RELEASE = $(PATH_DEVROOT)\Release
!if "$(DEBUG)"=="1"
PATH_TEMP = $(_T_PATH_TEMP)\Debug
PATH_RELEASE = $(_T_PATH_RELEASE)\Debug
!else
PATH_TEMP = $(_T_PATH_TEMP)\Retail
PATH_RELEASE = $(_T_PATH_RELEASE)\Retail
!endif
############################################################################# # \DevRoot\Build\Include\Shell.mk # Include Makefile for automated build process. # Defines shell commands used in the build process. # (c) Aspi Havewala 1998 ############################################################################## !if "$(VERBOSE)"!="1" CMDECHO = @ !endif DISPLAY = $(CMDECHO)echo IFF = $(CMDECHO)if COPY = $(CMDECHO)xcopy /S /E DELETE = $(CMDECHO)del /q DELTREE = $(CMDECHO)rmdir /S /Q MAKEDIR = $(CMDECHO)mkdir CC = $(CMDECHO)cl /nologo LINK = $(CMDECHO)link /nologo BROWSERBUILDER = $(CMDECHO)bscmake /nologo
############################################################################# # \DevRoot\Build\Include\Options.mk # Include Makefile for automated build process. # Defines options that modify the build process. # (c) Aspi Havewala 1998 ############################################################################# # Specify compiler options # Compile only - /c # Name temporary file - /Fo CFLAGS = /c /Fo$*.obj # If browser option off # Name browser file - /Fr !if "$(NOBROWSE)"!="1" CFLAGS = $(CFLAGS) /Fr$*.sbr !endif # If debug build, # Turn off optimizations - /Od # Generate debug info - /Zi # else # Enable global optimization - /Og !if "$(DEBUG)"=="1" CFLAGS = $(CFLAGS) /Od /Zi !else CFLAGS = $(CFLAGS) /Og !endif # Define a generic line for compiling a source file COMPILEME = $(CC) $(CFLAGS) $** # Specify linker options # Name output file - /OUT: LFLAGS = /OUT:$@ /LIBPATH:"$(MSDEVDIR)"\..\Vc\Lib /PDB:$*.pdb /MAP:$*.map !if "$(DEBUG)"=="1" LFLAGS = $(LFLAGS) /DEBUG !else !endif # Define a generic line for linking LINKME = $(LINK) $(LFLAGS) $** # Define a generic line for building a browser database !if "$(NOBROWSE)"!="1" BUILDBROWSERDB = $(BROWSERBUILDER) /o $*.bsc $(PATH_TEMP)\*.sbr !endif
##############################################################################
# \DevRoot\Build\Include\Targets.mk
# Include Makefile for automated build process.
# Defines rules for stock targets for the build process.
# (c) Aspi Havewala 1998
##############################################################################
# Rules for building temporary and release directories.
Setup: $(PATH_RELEASE) $(PATH_TEMP)
$(PATH_RELEASE): $(_T_PATH_RELEASE)
$(IFF) not exist $@ $(MAKEDIR) $@
$(_T_PATH_RELEASE):
$(IFF) not exist $@ $(MAKEDIR) $@
$(PATH_TEMP): $(_T_PATH_TEMP)
$(IFF) not exist $@ $(MAKEDIR) $@
$(_T_PATH_TEMP):
$(IFF) not exist $@ $(MAKEDIR) $@
# Rule to clean up temporary files.
Clean::
$(IFF) exist $(PATH_TEMP) $(DELTREE) $(PATH_TEMP)
Usage::
$(DISPLAY) "nmake [Targets]
[Options] "
$(DISPLAY)
"Targets "
$(DISPLAY) " All: Builds all
targets "
$(DISPLAY) " Clean: Cleans all output
files "
$(DISPLAY) " Usage: Displays this
message "
$(DISPLAY)
"Options "
$(DISPLAY) " VERBOSE = 0/1: Displays trace for build
process "
$(DISPLAY) " NOBROWSE = 1/0: Disables building of browser
database "
$(DISPLAY) " DEBUG = 0/1: Builds for retail or
debug "
############################################################################## # \DevRoot\Build\Include\Vc.mk # Include Makefile for version control integration. # Defines variables for version control command line and flags # (c) Aspi Havewala 1998 ############################################################################## # Version Control command line to refresh all source. VC_CMD = -"$(MSDEVDIR)\..\Vss\Win32\Ss.exe" Get $$/ # Version control flags. Note that this requires a special user called # Builder to be created by the Version Control administrator. VC_FLAGS =-I-Y -R -YBuilder -GTU # If clean refresh, specify additional flags. !if "$(CLEAN)"=="1" VC_FLAGS = $(VC_FLAGS) -GWR !endif
##############################################################################
# \DevRoot\AlarmClock\Makefile
# Master Makefile for automated build process.
# (c) Aspi Havewala 1998
##############################################################################
# Define a path for the development root
!ifndef PATH_DEVROOT
PATH_DEVROOT = \Aspi's~1\Articles\Versio~1\DevRoot
!endif
PATH_BUILD_INCLUDE = $(PATH_DEVROOT)\Build\Include
# Set up home directory in predefined variable
PATH_HOME = $(PATH_DEVROOT)\AlarmClock
!include <$(PATH_BUILD_INCLUDE)\Paths.mk>
!include <$(PATH_BUILD_INCLUDE)\Shell.mk>
!include <$(PATH_BUILD_INCLUDE)\Options.mk>
!include <$(PATH_BUILD_INCLUDE)\Vc.mk>
# Specify additional compiler flags
# Optimize for Windows application - /GA
# Add new directories to include path = /I
CFLAGS = $(CFLAGS) /GA /I$(PATH_HOME)\Common\Include
# Specify additional linker flags
# LFLAGS = $(LFLAGS) ...
All: Setup $(PATH_RELEASE)\AlarmClock.exe
# Rule for final executable.
$(PATH_RELEASE)\AlarmClock.exe: \
$(PATH_TEMP)\Timer.obj \
$(PATH_TEMP)\Date.obj \
$(PATH_TEMP)\Time.obj \
$(PATH_TEMP)\SetDate.obj \
$(PATH_TEMP)\SetTime.obj \
$(PATH_TEMP)\Sound.obj
$(LINKME)
$(BUILDBROWSERDB)
# Rules for compiling individual source files.
$(PATH_TEMP)\Timer.obj: Alarm\Timer.cpp
$(COMPILEME)
$(PATH_TEMP)\Date.obj: DateTime\Date.cpp
$(COMPILEME)
$(PATH_TEMP)\Time.obj: DateTime\Time.cpp
$(COMPILEME)
$(PATH_TEMP)\SetDate.obj: UserInterface\SetDate.cpp
$(COMPILEME)
$(PATH_TEMP)\SetTime.obj: UserInterface\SetTime.cpp
$(COMPILEME)
$(PATH_TEMP)\Sound.obj: UserInterface\Sound.cpp
$(COMPILEME)
# Clean up all output files.
Clean::
$(IFF) exist $(PATH_RELEASE)\AlarmClock.exe $(DELETE)
$(PATH_RELEASE)\AlarmClock.exe
# Get latest copy of source code from version control.
Refresh:
$(CHANGEDIR) $(PATH_DEVROOT)
$(VC_CMD) $(VC_FLAGS)
!include <$(PATH_BUILD_INCLUDE)\Targets.mk>