 Platform-Independent Stress Testing from the Command Line
Alan M. Berg
This article will describe the use of various open
source tools for scripting without any specific platform dependences.
Specifically, I will show how to plumb the Ant [1] and Jmeter [2] tools
together via the command line, generating HTML reports. I will also show
how to use email, scp, and file renaming. This information should allow you
to take the coding further and apply Ant to
other problem domains. Platform-independent scripting is a very handy tool
in a systems administrator's arsenal.
Ant is a task-based tool that reads in an XML
file(s), known as a build file, and performs specific tasks based on the
build. Example tasks include the archiving of directory structures,
emailing, secure copying, compiling token replacement, deploying Web
applications, etc. Ant was originally billed as a tool to take over the
role of the make command.
However, Ant has spread its roots further to encapsulate whole realms of
functionality. The quick growth of tasks can be attributed to the fact that
Ant is easily extendable. Tasks are written in Java and can be enabled via
the build file.
Ant's built-in features allow the following
types of tasks:
1. Archiving and file. These tasks include: bzip,
zip, jar, war, copy, delete, mkdir.
2. Programming tasks, such as javac, cvs.
3. Remote tasks, such as FTP, Rexec, Scp, mail, sql.
4. Logging, such as the record task.
5. Extensions. Ant tasks are Java classes that are
relatively easy to write.
There are numerous examples of contributed tasks; we
will take advantage of two of these: the Jmeter and Ant contribution
extensions. Note that certain operating system features do exist in Ant,
specifically file permission and exec tasks. Please be careful to check
these tasks against your target deployments.
Jmeter is a stress-testing and assertion tool. Jmeter
has a GUI for building test plans and can be run from the command line or
via an extended Ant task. A typical PC can run 500-1000 concurrent threads
sending HTTP requests. It is worth noting that for a receiving system a
thread can be more active than a typical user. Five hundred threads are
probably more like 1500 real users. Jmeter can fire off requests to http,
ftp, jdbc (database), soap (http with xml mixed in), LDAP, JMS (message
queues), and mail. This list of target protocols is a typical mix for
enterprise-wide infrastructures.
Ant, Jmeter, Tomcat, and Apache have been developed
under the Apache project [3] umbrella. The Ant and Jmeter tools are written
in Java and, thanks to the JVM, can run on any Java-enabled operating
system. The supported systems include Solaris, Linux, AIX, HP-UX, BSD,
Windows, and Apple.
Stress-Testing Basics
So, you want to hit your systems hard and see what
happens. Stress testing is an excellent tool for gaining confidence in a
given infrastructure's ability to survive the onset of the end user.
In general, there are three scenarios. The first is the acceleration of a
burn period for new hardware. On one occasion, I had the pleasure of seeing
a system fail five times all due to motherboard issues. By the application
of a focused stress test, the motherboard issue would have been found
before the hardware had reached production status. Thus stress testing
would have avoided a significant number of help desk calls from irate
clients.
The second scenario is that of knocking systems that
have been patched. Before deploying a patch to production, we first update
an acceptance environment and, if required, again hit the systems hard.
Logging is vigorously analyzed for those strange exception numbers that
occur when systems start to fail. The third and most difficult to achieve
situation is that of both stress testing and performing functional tests
within a full software cycle. Figure 1 defines one possible solution.
Automated scripts run tests against development,
acceptance, and production. The scripts are called via cron jobs or
scheduled tasks, and the generated html-formatted results are sent by scp
to a directory on a results server. Emails are sent to a distribution list
of interested parties. The advantage of automation is the capturing of
bogus code, patch, or intermittent errors early on in the cycle.
Furthermore, this process provides you, as a systems administrator, with
the feeling of orderly control over a potentially chaotic environment. By
the end of this article, you should have enough background knowledge to be
able to consider the pros and cons of building this structure based on Ant.
What Is to Come?
The source code accompanying this article contains
four example build files. The first example sends email and has the purpose
of introducing you to the build file structure. The second example archives
log files from a directory structure and sends the archive by email. The
purpose of the second example is to show how instantaneous it is to script
for complex tasks. Further, the example highlights the relevance of using
Ant in your environment. The third example is a script that runs a stress
test automatically. This example shows how to extend Ant for extra tasks
and how to perform basic stress tests. The final example performs a series
of stress tests. Here you will learn about iteration through tasks and
passing of variables.
Setting Up Your Environment
The following scripts will work with Jmeter 2.1 or
above and Ant 1.6 or above. Both packages are downloaded as zip files [4,
5] that once unzipped can be directly run. Java 1.4 is required. Please
follow the simple installation instructions contained within the packages.
To run Ant from a cron job, I typically set a couple
of environment variables in one file and call the file from a batch file.
For example:
file: setenv
export CVS_RSH=/usr/local/bin/ssh
export JAVA_HOME=/usr/java
export ANT_HOME=/opt/usr/ant
export CLASSPATH=/opt/usr/ant/lib/mail.jar:/opt/usr/ant/ \
lib/activation.jar
export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH
cronjob batch
#!/usr/bin/bash
source /opt/usr/scripts/set_env
ant -f /opt/usr/scripts/example1.xml ?Dtest.value=45
Note that if you want to extend Ant tasks, you would
normally set the extra libraries in a location defined in the CLASSPATH.
Further, if you want to pass a value to the build files from the command
line, then you would need to follow the convention of ant -Dname=value. Finally, Ant tasks
obviously have a lot of power; delete dir, for example, has the same effect as rm -rf. Therefore, it would not be a
good idea to run as a privileged user.
Project Basics
Before going further, let's define some basic
terminology -- a project defines the whole. A target is a collection
of tasks that are grouped together. One target may be to zip all files and
then copy the archive to a backup mount; another may be to initiate the
directory structure or delete it. Therefore, a project may contain a series of targets. A target in itself may contain a group
of tasks. The build file has an XML structure and the tags either are
self-contained (e.g., <fileset />) or may enclose other tags (e.g.,
<fileset></fileset>).
Example 1
Listing 1 enables Ant to send an email. The basic structure is as follows:
Project
Define some properties
Target
Do some tasks
Target
Do more tasks
Translating project into Ant syntax:
<project name="example1" default="sendmail">
The code fragment states that if we run Ant from the
command line, then the sendmail target is called by default. To run another
task first, simply add the target name after Ant (e.g., ant targetname). To
obtain the full list of targets in a given build file, type ant -projecthelp.
The <property file=
"default.properties"/> statement is powerful; it tells Ant
to load in the variables from the property file default.properties. The
property file has the syntax value=name (e.g., jmeter.host=localhost). To
reference the variable within Ant, you would need to enclose the variable
with ${} (e.g., ${Jmeter.host}).
The mail task looks pretty straightforward apart from
the fileset attribute. This attribute is common to many tasks:
<fileset dir="${log.dir}">
<include name="**/*.zip"/>
</fileset>
In other words, the fileset is the set of files under
the ${log.dir} directory and its children as defined by log.dir= in the
property file. The set of files should only include zip files.
The mail task uses the fileset attribute to define the
files to be sent by mail. The task has a dependency on outside libraries.
Download the mail and activation.jar files [6, 7] and add their locations
to the CLASSPATH variable.
Example 2
Listing 2 copies all .log files from a given directory
structure and places the logs in an archive. For example, the time stamp is
placed in the variable today:
<tstamp prefix="today"/>
There are three formats for the time stamp DSTAMP
(yyyMMdd), TSTAMP(HHmm) and TODAY (MMMM dd yyyy). To reference the DSTAMP
format, we need to use the notation ${today.DSTAMP}.
Copying files is straightforward; however, you may
want to map the names or directory structures. Ant can achieve this in
multiple ways, such as via mapping(s) within the fileset:
<chainedmapper>
<flattenmapper/>
<globmapper from="*.log" to="*_${today.DSTAMP}.log" />
</chainedmapper>
The "chainedmapper" is used to make a
composite of other mappers. The "flattenmapper" mapper removes
the path information, and the "globmapper" does a simple
wild-card mapping and replacing, in the above situation adding a timestamp
to the log filenames. For more complex mappings, I would recommend looking
at the "regexp" mapper.
Zipping the archive is straightforward:
<zip zipfile="${log.dir}/logs_${today.DSTAMP}.zip"
basedir="${log.dir}/${today.DSTAMP}"/>
The sending of email occurs by setting the
mail.message property and calling example1.xml:
<property name="mail.message" value="Have a nice day
${today.TODAY}" />
<ant antfile="example1.xml"/>
Notice that mail.message is not defined in the
default.properties file. Otherwise, the default.properties setting will
override the value set.
The Plan
Note that the important theme of this article is the
control and iteration of tasks via Ant and that Jmeter stress testing is
just one example. If you feel that the details of the Jmeter test plan are
not immediately relevant, feel free to skip the next couple of paragraphs.
Like Ant, Jmeter uses an XML configuration, known as a
test plan, to define which tests to perform. However, unlike Ant, the plans
generally are made via a GUI. In this exercise, we will run a templated
test plan. The Ant script will copy the test plan and replace information
such as target host and port with parameters stored in the
default.properties file. The templated test plan is found under
examples/Jmeter/templates/plan.jmx and is best viewed from the GUI. A
screen grab is shown in Figure 2. Notice the @PORT@ string. This is one of
the tokens that will be stamped.
The plan consists of a number of elements. The top
level is a thread group, which defines the number of threads to be run. The
child HTTP cookie manager remembers the cookie per thread and returns it if
asked by the server. The HTTP user parameter modifier replaces specific
parameters in an HTTP request with data from an XML file. This is useful,
for example, if you want to log into a server with the parameters user and
password.
The data file has an XML structure and needs also to
sit under the bin directory of the Jmeter instance. The final component is
the HTTP request, which sends a request per thread to a server. Notice the
parameters user and pass are left blank and will be filled in from the data
file. For convenience, a very brutal Perl script (scripts/make_user.pl) is
supplied to generate a data file. Otherwise, you may use the file found
under examples/Jmeter/user_data.xml.
Initial Conditions
Setting up the test environment requires installing an
example war file under a Servlet container, such as Tomcat or JBoss. The
war file can be built by running at the top level of the source code Ant
build, then collecting build/war/Jmeter_target.war and deploying. The
application is composed of two pages. Index.html is a login form, and
parser.jsp sends a message based on the user name and password. The
advantage of JSP pages is that you can change the pages on the fly without
rebuilding the application.
To define a new Jmeter task in Ant, download the
Jmeter library [8]. Add the location of the library to the CLASSPATH.
Example 3
Listing 3 is responsible for running the Jmeter task.
However, workflow is required so that we can run the task more than once
without overwriting the results or ruining the template. The flow is as
follows:
1. The template is copied to a second directory, and
the @@ tokens are replaced with values defined in the default.properties
file.
2. The test plan is carried out, and the results are
saved. The results name is made unique by following a convention that
includes the parameter used and a time stamp.
3. An XSL task is run that turns the results into a
statistics page.
Within the listing, a new taskdef is needed to allow
us to apply the Jmeter task:
<taskdef
name="jmeter"
classpath="../extlib/ant-jmeter.jar"
classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>
Next, we need to copy the templated test plan to a
second directory, replacing the tokens with variables in the properties
file. This is achieved by:
<filter token="LOOPS" value="${example.three.loops}"/>
<copy todir="${jmeter.workpad}" filtering="true" overwrite="true>">
Note that overwrite is set to "true" and
that the token @LOOPS@ in the test plan will be replaced with the variable
example.three.loops from the properties file.
The Jmeter task runs Jmeter and the XSLT task
generates the html report. Figure 3 is a screen grab of one such report.
A More Advanced Project
We will use the excellent Ant-contrib library, which can be found at:
http://www.jcraft.com/jsch/index.html
Please download the library and add to the CLASSPATH.
Also, please comment out in default.properties the property
example.three.threads.
Example 4
Ramp testing is stress testing with different levels
of usage. One of the motivations behind ramp testing is to give an
indication of how a system performs under future loads. Figure 4 shows two
typical responses to a ramp test. The linear response under load is exactly
what you want to see. If a production system is successful and more users
wish to visit the application then, with a linear response curve, you will
have more time to plan for replacement or additions to your hardware farm.
However, with the second response curve, you will have a significantly
condensed period to react in. Further, instabilities in response may
confuse the underlying cause. In principle, it is better to find out the
limitations of your system in acceptance than in production. It costs in
short term effort, but generally saves in the middle term.
Ant is somewhat weak in mathematics and iteration,
the task for list processes the example.four.thread property and expects a
comma-delimited string (e.g., example.four.threads=5,10,15,20,30,40,50). Each element mentioned within <for
</for> will be looped through.
<for list="${example.four.threads}" param="threads">
The <sequential> statement forces the code to be run sequentially. If
you wish you can also run the tasks in parallel. But that action risks side
effects such as race conditions and resource deadlocks.
Within the for loop, we call the antfile example3,
which runs an individual Jmeter testplan. The
parameter example.three.threads is set to the
iterated value:
<ant antfile="example3.xml" target="JmeterTest"
inheritall="true" >
<property name="example.three.threads" value="@{threads}"/>
</ant>
Note that you will need to comment out the
example.three.threads in the property file, or the properties file value will win over the iterated value. Also note the
strangeness with the "@" instead of the "$" for
threads; you will need to accept this for the task at hand. Sometimes life
is just like that.
Now that I've run through the basics, here are a
few hints based on my own experience:
1. Make paths relative. This practice will
allow you to keep resources within your build structure and make transport
of the structure to other machines easier.
2. Make your targets as brief and as discreet
as possible. You should call one target from another.
3. Always run as a non-privileged user or some
of the more powerful tasks may cause pain if misconfigured.
4. Resist the temptation to use OS-specific
code such as exec. If required, then move into a separate task.
5. Add meaningful descriptions so that other
systems administrators can guess what is going on.
6. Try to collect often used targets into
generic build files.
7. Use Echo copiously.
8. Make sure property names are consistent
between the build and property files.
9. Be careful when you overwrite files or
clean directories as you will be indirectly affecting other tasks.
10. Do not get set in your ways. Find excuses to try
out new tasks, and this will allow you to expand the applicability of the
tool further.
11. Testing should be done from another server as you
may affect the tester by the resources used by the tested.
12. Let the Jmeter GUI do the configuration work for
you.
Looking back to the infrastructure described in Figure
1, we have cycled through all the relevant tasks and approaches from the
Ant side apart from the scp task. Code for this would look similar to:
<scp todir="${scp.user}:${scp.pass}@${scp.host}:${scp.rootdir}" \
trust="true" >
<fileset dir="${clean_dir}/scp_final"/>
</scp>
The task requires the scp dependency library [9].
That brings us to the assertion tests. Jmeter has
components for assertion testing. If a particular string or lack of a
string is returned from a response, then an error can be raised. Therefore,
to render assertion tests from Ant requires no more scripting skills. It is a question of making the correct test plans
and templates. However, that said, anteater is
an Ant-specific assertion task. If you would like a more direct approach,
then this contribution should be your first port of call. Other tasks of
interest include ones to deploy and maintain Tomcat and the standard SQL
task to connect and manipulate databases. They are many more tasks worth
exploring.
Enjoy your experimentations. I wish you luck.
Resources
1. Ant Home -- http://ant.apache.org/
2. Jmeter home -- http://jakarta.apache.org/jmeter/
3. Apache home -- http://www.apache.org
4. Ant download page -- http://ant.apache.org/bindownload.cgi
5. Jmeter download page -- http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi
6. Mail library download page -- http://java.sun.com/products/javamail/
7. Activation download page -- http://java.sun.com/products/javabeans/jaf/index.jsp
8. Jmeter library download page -- http://www.programmerplanet.org/ant-jmeter/
9. Scp dependency library -- http://www.jcraft.com/jsch/index.html
Alan M. Berg has been a lead developer at the Central
Computer Services at the University of Amsterdam for the past seven years.
He likes to get his hands dirty with the building and gluing of systems. In
previous incarnations he was a technical writer, an Internet/Linux course
writer, and before that a science teacher. He remains agile by playing
computer games with his kids (Nelson and Lawrence) who sadly consistently
beat him.
|