Java Solutions

The Tattletale Technique

Toby Reyelts

With this technique and a new 1.3 feature, finding resource leaks has never been so easy.

Basic Resource Management

Practical experience has taught many Java developers one thing: critical resources (mutexes, database connections, transactions, file handles, etc.) require timely and systematic release. Unfortunately, Java’s garbage collector is not up to that job. According to the Java Language Specification, there are no guarantees when a garbage collector will run, when it will collect an object, or when it will finalize an object — if ever. Even more unfortunately, Java’s counterpart to the C++ destructor (the finally block) is both tedious and error-prone, requiring developers to constantly remember and duplicate resource-releasing code. Consequently, even good Java developers can forget to release critical resources.

There is a light at the end of the tunnel. Java may make it easier to leak critical resources, but it also provides the necessary mechanisms to easily track them down. The Tattletale technique is a simple method for designing new classes and retrofitting existing classes to quickly and easily detect the offending code responsible for leaking resources.

A Typical Scenario

You are a senior engineer on your company’s software engineering team. You’ve been called to a customer site because their server has been habitually hanging. You find the following error message scattered throughout the server’s log file:

Connection Pool Timeout: None of the pooled connections are available for use.

The problem is obvious. At least one place in the application code is leaking database connections. The real question is, where? You don’t have the time to search through hundreds of thousands of lines of code, but what other choice do you have? You decide to use the Tattletale technique to track down the code that is forgetting to close the connections.

The first step in implementing the technique is to discover where the client is allocating the resource. In this case, your team may have written code with bugs in it, but that doesn’t mean they don’t understand good design. They’ve created a factory to centralize the retrieval of database connections, and that is where the clients are allocating their resources.

class ConnectionFactory {

  Connection getConnection() {

    // return allocate();

  }

}

The next step is to figure out which clients are allocating which resources. Java can tell you which client code is calling the getConnection method through a stack trace. You also can get a stack trace by creating a new Throwable. Now, your only problem is associating stack traces with Connections. You decide that the most sensible way to do that is to embed a Throwable with Connection. So, you create an implementation of the Connection interface that allocates a new Throwable upon creation. Your Connection implementation isn’t anything special — it just delegates to a real Connection.

class TtConnection
  implements Connection {

  private Throwable trace;
  private Connection contained;

  TtConnection( Connection c ) {
    contained = c;
    trace = new Throwable();
  } 

  // Other methods required to
  // implement the interface
  // with methods that do nothing
  // but call the similarly named
  // method in "contained"
}

Now, when clients call ConnectionFactory.getConnection, you return them a new TtConnection that wraps the real underlying Connection.

class ConnectionFactory {
  Connection getConnection() {
    // Connection c = allocate();
    return new TtConnection( c );
  }
}

Finally, the last step is to detect when a client actually “leaks” the resource. In order to release a connection, the client must call its close method. The garbage collector doesn’t collect objects until they’ve become unreferenceable. Given that knowledge, you update TtConnection so that it prints out a message if it has become unreferenceable and hasn’t been closed:

class TtConnection
  implements Connection {

  private boolean hasBeenClosed = false;
  private Throwable trace;
  private Connection connection;

  TtConnection( Connection c ) {
    connection = c;
    trace = new Throwable();
  } 

  public void close()
    throws SQLException {
    hasBeenClosed =  true;
    connection.close();
  }

  public void finalize() {
    if ( ! hasBeenClosed ) {
      System.out.println(
      "The client did not close " +
      "a connection created at " );
      trace.printStackTrace();
      // Close connection here
    }
  }

  // Other methods required to implement the interface 
  // go here.
}

With that taken care of, you recompile your code and re-deploy your application. Lo and behold, you start receiving stack traces pointing directly to the client code that has been leaking connections. With a quick edit and recompile, you fix the bug and save the day, all in a matter of minutes. Your company rewards you with a large bonus. (Okay, maybe not.)

A Generic Tattletale

Perhaps there are a lot of different types of critical resources that you would like to track down in the case of a leak. Unfortunately, implementing all of the required classes might take a significant amount of time and effort. Since the implementation doesn’t really vary between the different types of resources, it seems like there should be some way to reuse code — and there is.

Starting with Java 1.3, Sun introduced a new Proxy class in the java.lang.reflect package, which can be used to create dynamic proxy classes at run time. These dynamic proxy classes allow you to insert crosscutting (fundamentally common) behavior into otherwise unrelated classes, in a manner very akin to aspect-oriented programming. In this case, it is the Tattletale behavior that you’ll be inserting into orthogonal classes.

First, decide how the generic Tattletale will determine when its resource has been released. Generally speaking, most resources have a single public method that is used to release the resource (e.g., close or dispose). So, the generic Tattletale will mark an object as released if a specified method has been called.

Next, actually codify this behavior. To create a dynamic Proxy, it’s necessary to specify two pieces of information: 1) the set of interfaces that the Proxy will implement, and 2) the InvocationHandler for the Proxy. Proxy works by delegating all of the method calls on the interfaces that it implements to InvocationHandler. This is where the logic for Proxy has to be implemented. In the Tattletale implementation of InvocationHandler (see Listing 1, TtHandler.java), the logic very closely mimics that of TtConnection. Upon construction, the handler stores away a stack trace. The only difference in construction is that the handler also requires the name of the method that cleans up the resource. The method handling is also the same as TtConnection. If the user calls any method, it is delegated back to the object that is being wrapped by InvocationHandler. If the user calls the method that releases the resource, the handler sets a flag. Finally, the implementation of finalize is also the same as TtConnection. If the resource-release method has not been called, then the handler prints a trace to the client that allocated the resource. That’s all there is to it.

For example, to use the generic Tattletale instead of TtConnection, you can change ConnectionFactory as follows:

class ConnectionFactory {

 static interfaces = new Class[] {
   Connection.class
 };

 Connection getConnection() {
   // Connection c = allocate();
   return (Connection)
Proxy.newProxyInstance(
     c.getClass().getClassLoader(),
     interfaces,
     new TtHandler( c, "close" )
   );
 }
}

Tracking Classes That Can’t Be Subclassed

Up to this point, I’ve only demonstrated how to track resources that have interfaces (like Connection). It’s fairly trivial to extend the Tattletale technique to work with any resource that can be subclassed. For resources that can’t be subclassed, either because they are final or have private constructors, it’s a non-trivial task to make the Tattletale technique work. In fact, it’s not possible to make the technique work without actually modifying the resource class.

The basic problem is that it’s impossible to tell when a client leaks the resource without explicit cooperation from the resource. One might be tempted to use the Reference and ReferenceQueue classes from the java.lang.ref package, but they can’t detect the leak either. References and ReferenceQueues work well to determine when an object has become weakly referenceable or eligible for finalization. However, once the virtual machine notifies you of the change in the reference status, you can no longer reference the original object to determine whether it was properly released. RefTest.java (available for download at <www.cuj.com/java/code.htm>) contains an application that demonstrates this problem. Although the virtual machine does clear the reference in MutexRef and place it onto the ReferenceQueue, it does this without calling MutexRef.clear. In addition, once MutexRef has been placed on the ReferenceQueue, the reference to the Mutex object has been cleared, as demonstrated by the output of the program: "Mutex reference: null". Finally, MutexRef can’t keep an internal reference to Mutex, otherwise Mutex won’t be collected.

Besides actually changing the source code for the resource class, which is often not an alternative, one possible option for implementing the Tattletale technique in this case is to modify the bytecode in the resource class. For example, the free and open-source Byte Code Engineering Library available at <http://bcel.sf.net> provides an intuitive API for reading and modifying Java bytecode. In fact, it comes with several examples for reading in class files and making non-trivial changes to the bytecode instructions in method definitions.

Shortcomings

The Tattletale technique has a few shortcomings.

The main problem with Tattletale as implemented is that there is no absolute guarantee that a garbage collector will finalize an object. The old Runtime.runFinalizationOnExit method has been deprecated (because it just didn’t — and can’t — work in a multithreaded environment). One alternative is not to use finalizers at all. Rather, the constructor can create a Runnable object that checks whether or not the object has been closed and then can pass that object to Runtime.addShutdownHook. This way the check is guaranteed to occur. Although the Java Language Specification doesn’t make any promises about garbage collection, practically speaking, all of the existing implementations work more than well enough to make this technique useful.

A second weakness is that it can be tough to make the technique work if clients do not allocate their resources from a modifiable, central source. For example, if clients allocated database connections directly using DriverManager.getConnection, you couldn’t alter the DriverManager source to return your own special Connection implementation (although you could, with more effort, implement a Driver wrapper).

Conclusion

The design decisions made in the implementation of Java can make it difficult to consistently release critical resources. The Tattletale technique makes it easy to track down and eliminate resource leaks.

Toby Reyelts has been working with Java since before its 1.0.2 release and with C++ circa the initial ARM release. His work ranges from low-level JVM integration and JNI implementation to high-level distributed application design. He is currently a software architect responsible for the overall design and implementation of custom enterprise software solutions, primarily built upon the J2EE. He graduated from Georgia Tech with a bachelor’s degree in computer science degree and a specialization in operating systems and digital systems design and implementation. With the little free time he has, he likes to research hardware, operating systems, and computer languages.