Dr. Dobb's Journal August, 2004
What characteristics differentiate Java servlets from other types of programs? How do these characteristics affect the manner in which you go about testing servlets? If you're going to test servlets, then you're going to have to be able to answer questions such as these.
In considering servlets and how to test them, it's useful to review some fundamentals about what makes a servlet a servlet. The Java Servlet Specification (http://java.sun.com/products/servlet) defines a servlet as a:
...Java technology based web component, managed by a container, that generates dynamic content. Like other Java-based components, servlets are platform independent Java classes that are compiled to platform neutral bytecode that can be loaded dynamically into and run by a Java enabled web server...
The test implications of servlets being managed by containers are that you have to be concerned not only with how your servlet handles longevity (you want your servlets to run unattended for extended periods); scalability (you want the application supported by your servlets to be able to grow to meet increased usage); integration with other software (such as databases to support dynamic content); and security.
Moreover, running applications through web servers supports distributed application configurations, where components can be hosted on multiple physical servers. Being able to load applications dynamically to be run by a web server makes down time a thing of the past. The Internet and Web mean that you can no longer assume that 2:00 am your local time zone is a quiet time when you have customers everywhere from Boston to Belarus. The test implications here are the need to verify both distributed configurations and the ability of the servlet to run unattended over time.
With this in mind, servlet-specific software test considerations you should keep in mind in designing and executing tests for servlets can be divided into these logical groupings:
One factor that can increase a web application's reliability is hosting it on multiple web servers, instead of on a single server. If one server goes out of service, other servers can continue to respond to clients' requests. You can see a simple illustration of this distributed approach if you use the nslookup command to retrieve the IP addresses of a well-known, high-capacity web site. From a user perspective, the distributed architecture in Figure 1 is seamless. Users enter the same URL, and their requests are handled. From a testing perspective, however, there are functions that you have to think about.
Maintaining persistent data and state. In preInternet days, a major aspect of testing an application built on distributed servers involved verifying that once clients made connection to servers, the connections would be preserved by fault-tolerant mechanisms if a server failed. With servlet-based web applications built around the HTTP protocol, you never have a persistent connection because the protocol is stateless. When you access a content-rich web site such as CNN or use a web application, your view is that you have fixed, dedicated connection from your client to the server. That view, however, is an illusion. The single "click" that displayed content in your browser actually sets in motion a potentially large number of discrete HTTP GET requests.
When you're testing servlet-based web applications, it may be the case that the "servlet environment" actually consists of instances of the servlet running on multiple physical servers. Configurations such as this ensure that applications will handle higher traffic than a single physical server and removes the server as a single point of failure. However, it also means that the application cannot maintain persistent data on a servlet's server. If the servlet you're testing supports a distributed web application, then it may maintain state via Java HttpSession objects API to store user information in session objects. These objects have to be made persistent by storing them in a central resource that is available to all instances of the servlet; for example, on an application or database server.
With this in mind, the sort of tests you should perform include continuous session maintenance, which gives you the opportunity for some negative test scenarios. For instance, you can have test clients open sessions via servlets on one server, then halt the web server or servlet container on that server and verify that the users' sessions are maintained when they continue accessing the application via servlets on other servers.
Coping with outages. One of the goals of any distributed architecture is to avoid a single point of failure. Distributing an application's servlets across multiple physical servers can help do this, but there may still be single points of failure in the application's middle (application servers) or EIS tiers.
What sorts of tests should you perform? This is another opportunity for negative test scenarios. You want to verify how the servlet under test reacts to either not being able to make or maintain a connection to its database or application server. Unlike the stateless client connections, these connections must be persistent. Depending on the details of the application's design, you may see configurable retry algorithms. Regardless of the application's design details, you should check that informative error messages are returned to clients in case resources needed by the servlets are not available.
Handling logging and logs. If the servlets and their servlet containers are running on multiple servers, then their corresponding logs are either being generated on multiple servers or are being written to multiple files/directories on some shared file system. Either way, logging information ends up in more than one place. The servlet implementation can organize things that are explicitly generated by the servlet, but you'll also have to deal with the logs generated by the servlet's web server and servlet container. This distribution of logging can complicate efforts to debug problems as it makes it difficult to trace through a specific sequence of a client's interactions with the servlet—and the application has to be able to manage the logs on multiple servers.
What sorts of tests should you perform in this case? If you're planning on implementing a distributed, servlet-based application, then you'll have to think about some mechanism for collecting and parsing logs from multiple sources. Ideally, this mechanism includes a means to merge multiple logs together so that you can follow a client's actions as it accesses multiple servlets. Tests for this type of mechanism can be divided into these categories: data collection (are the logs collected?), results calculation (are the logs merged and analyzed correctly?), and report generation and distribution (are reports created and distributed?).
One of the more mundane aspects of writing out large numbers of logs is that the log files must be managed or you'll run out of space. Testing for this is relatively easy—just fill up the disk and see how the servlet reacts when it tries to write to its log. A better approach is to have a utility (perhaps another servlet) monitor free disk space and proactively delete/archive old logs or send an alert to sysadmins.
The J2EE platform provides for policy-based, configurable "fine-grained access" control (http://java.sun.com/features/1999/
12/j2ee.html). Whenever programs (including servlets) are loaded, they are assigned a set of permissions based on the security policy currently in effect. Each permission is tied to and enables access to a particular resource (being able to access a specific host and port, for instance). One common approach is to use Access Control Lists (ACL) (http://java.sun.com/j2se/1.4.2/docs/api/java/security/acl/Acl.html), supported by the java.security.acl interface. ACLs are data objects that tie resources and permissions to principals (individual users and groups). The extent to which security can be customized for an application, coupled with the Java's built-in security features (exception handling, for instance) make it possible to create secure servlets. This doesn't mean you can forget about testing for security, however.
What aspects of security specifically affect servlets? It's likely that the application supported by the servlet under test supports various levels of user permissions. Some users require administrative-level permissions, while others will be limited to end-user permissions only. You'll want to verify that these levels of permissions are enforced for various users and groups. You'll also want to verify that both positive and negative permissions are enforced. The safest security policy is to disable all access by default, and then only enable access explicitly on a selective basis. A good negative test is to delete or otherwise disable users after they have logged in. The servlet should recognize that the user is no longer valid by requiring that the user be periodically reauthorized, even if a valid session object exists.
It's always possible for an application's security to be compromised, regardless of how well its security policy is designed. What you want to look for in testing is to both verify the security policy's design and how it is actually implemented. What sorts of tests should you perform in this case? Encryption can provide a good measure of protection for passwords and other data, so it's important to verify that the data that's supposed to be encrypted actually is. It's also important to verify that the encryption isn't inadvertently sidestepped. For example, encrypting passwords or other customer data used to establish a session is a good idea. Writing that same data in plain text to a log file is a bad idea.
It's often the case that servlet-based applications provide for remote administration via an admin servlet. This can be convenient, but dangerous. Permissions to enable access to this servlet must be strictly controlled. In addition to verifying these permissions, you should verify that the use of this servlet is tracked via an audit trail.
Remember that one of the characteristics that set servlets apart from other types of programs is that they run within a servlet container. Accordingly, any security flaws in the container directly affect the security of the servlet. These security flaws may take the form of actual functional bugs in the container, or they may be caused by an error in how the container has been configured. Then again, security flaws are sometimes caused by people simply doing foolish things such as running the container out of the (UNIX) root user account, instead of from a user account dedicated to the container (and configured with the permissions needed by the container, but no more).
John Donne's famous "No man is an island, entire of itself..." is true for software these days, too. It's rare to find a program that doesn't have to integrate with some other programs. Most software products are built with components supplied, at least in part, by third parties. Sometimes, this means open source or freeware, and sometimes partnering with another company. Either way, testing the integration of components, systems, servers, servlets, and the like is becoming an increasingly important category of testing.
When you integrate multiple systems, you are in effect building a common language to bind the systems together. The elements of this language include software dependencies, configuration data, internal and external process coordination and data flows, and event reporting and security. Integration testing ("aggregation testing" might be a better term) verifies the operation of these language elements and of the entire integrated system, and identifies the points of conflict between them.
The servlet under test will very likely interact directly with other server-resident programs, possibly including other servlets. In some respects, these interactions are no different than interactions between nonservlet programs. The distinguishing factor for servlets is that these interactions take place on and between server-based software.
When it comes to the types of tests to perform, you'll want to verify that the startup dependencies function correctly for the servlet under test. There are a couple of different levels of dependencies to consider. First, there are dependencies between the servlet container and programs upon which the servlet depends. A good example of such a program is a database server (process). The mechanics of handling this vary with the operating system. For PCs the dependencies are specified in the services definitions, while for UNIX systems it's done via startup scripts. There are also dependencies between servlets to be considered. The mechanics of handling this will vary with the servlet container being used. For Tomcat, it's handled via the <load-on-startup> directive.
Verifying dependencies is another good opportunity for negative testing in that you will want to determine what happens if programs, utilities, and so on, that the servlet is dependent upon are not in place. The servlet should handle the missing dependencies gracefully by logging (and displaying for users) informative error messages. These types of tests should be performed when the servlet is started and after the servlet is running, as you'll be exercising different code when the servlet tries to start up when a dependency is not in place and when it loses contact with that dependency while it's trying to service requests from clients.
Java's exception-handling mechanism gives servlets a head start on security and reliability. But what about exceptions that are encountered in nonJava programs with which the servlet under test must interact? What sorts of tests should you perform? Some time ago, I encountered this situation: The servlet under test was integrated with a legacy source-code control system written in C++. During the early stages of testing, we encountered problems where the servlet's container (Tomcat) simply exited with an error message of "Exception outside of JVM." As you can imagine, this error message wasn't really helpful in debugging the root cause of the problem. We added debugging statements to the servlet to let us better track what was happening before the exception was encountered. This additional debugging information, coupled with improvements in newer versions of the servlet container (we were a little out of date) let us isolate the failing function within the code with which the servlet was integrated.
The problem we had to deal with was that the operation we were attempting to perform within the legacy product was failing. If we attempted the same operation via that legacy product's command-line interface, we received an error message. The lesson here is that your tests for the servlet should include forcing errors within the other products/code with which the servlet is integrated. The errors should be trapped by the other product and gracefully passed back to the servlet and its client.
Another characteristic that sets servlets apart from other types of programs is that, unlike programs that you start up, run for a particular purpose, and then shut down, servlets are intended to run unattended for extended periods of time. Once a servlet is instantiated and loaded by its container, it remains available to service requests until it is either made unavailable or until the servlet container is stopped. Accordingly, you should include specific longevity-related classes of tests in planning tests for your servlet.
For servlets, the types of integrations and integration tests you'll want to consider include finding memory leaks and verifying scalability.
One of the most useful features of the JVM is its garbage collector, which analyzes memory to find objects that are no longer in use and deletes them to free up memory. In spite of the garbage collector's best efforts, however, you can still have memory leaks in Java programs. These can be especially problematic for servlets because the leaks can grow while the servlet runs.
How does the garbage collector determine if an object is in use? It looks for objects that are no longer referenced. A running Java program contains threads. Each thread executes methods. Each method references objects via arguments or local variables. Collectively, these references are referred to as a "root" set of nodes. The garbage collector works through the objects in each node, the objects referenced in each node, the objects referenced by those nodes, and so on. All other objects on the heap (that is, memory used by all JVM threads) are tagged as unreachable and are eligible for garbage collection. Leaks occur when objects are reachable, but are no longer in use.
What sorts of tests should you perform? In contrast to short-term functional tests, in this case, you want to verify long-term results. In effect, you'll want to simulate real-world conditions where the servlet under test will have to function 24/7. So you'll have to establish a level of traffic between test clients and the servlet. There are several tools (such as the IBM/Rational Robot testing tool) available to simulate user sessions. The client, however, is not your main concern. How the servlet reacts to a regular, sustained level of incoming requests is: You can monitor memory usage in several ways. The easiest way to start is simply by using utilities supplied by the operating system. On Windows, the Task Manager lets you view total system memory in use and also the memory being used by any process. On UNIX systems, similar tools such as sar (on Solaris), vmstat (on BSD-based UNIX flavors), or top provide you with some basic measurements on memory usage. You're looking for increases in memory usage by the servlet's container to handle the client traffic, where the memory is never released when the traffic is stopped. If you do locate a pattern or memory loss with these basic tools, then you can investigate further with more specialized tools such as IBM/Rational Purify testing tool to identify the offending classes/methods.
The term "scalability" generally describes how software under test can respond to an ever-increasing level of incoming traffic. A more subtle type of scalability testing that is more closely related to memory usage involves how a servlet is able to respond to requests when the contents of those responses are affected by the long-running nature of a servlet.
What sorts of tests should you perform? You'll want to examine and verify the mechanisms to deal with data that starts small, but grows as the servlet runs. For example, I recently worked on a servlet-based client-server application where one of the features supported by the client was the retrieval of log files that tracked operations executed by the servlet. Under functional testing, the retrieval worked without error. However, after the server/servlet had been running for several consecutive days, the lists of log files that the client could access, and the log files themselves, had grown so large that the servlet's performance was severely degraded when it had to extract, format, and deliver log information on a large scale back to the client. The initial design of the servlet returned the entire set of requested log information in a single, large vector of text strings. Ultimately, the design was altered to return the information one page at a time. This resulted in multiple requests to the servlet, but each request only resulted in a small response.
To successfully test servlets, your tests have to take into account that as servlets, they must take advantage of the features supplied by Java and must live within the limitations of the servlet model.
DDJ