Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

Reloading Java Classes 201: How do ClassLoader leaks happen?

Other Articles in the Reloading Java Classes Series

A video presentation “Do you really get class loaders” by Jevgeni Kabanov

From ClassLoaders to Classes

If you have programmed in Java for some time you know that memory leaks do happen. Usually it’s the case of a collection somewhere with references to objects (e.g. listeners) that should have been cleared, but never were. Classloaders are a very special case of this, and unfortunately, with the current state of the Java platform, these leaks are both inevitable and costly: routinely causing OutOfMemoryError’s in production applications after just a few redeploys.

Let’s get started. Recalling RJC101: to reload a class we threw away the old classloader and created a new one, copying the object graph as best we could:

reloading-object

Every object had a reference to its class, which in turn had a reference to its classloader. However we didn’t mention that every classloader in turn has a reference to each of the classes it has loaded, each of which holds static fields defined in the class:

classloader-refs

This means that

  1. If a classloader is leaked it will hold on to all its classes and all their static fields. Static fields commonly hold caches, singleton objects, and various configuration and application states. Even if your application doesn’t have any large static caches, it doesn’t mean that the framework you use doesn’t hold them for you (e.g. Log4J is a common culprit as it’s often put in the server classpath). This explains why leaking a classloader can be so expensive.
  2. To leak a classloader it’s enough to leave a reference to any object, created from a class, loaded by that classloader. Even if that object seems completely harmless (e.g. doesn’t have a single field), it will still hold on to its classloader and all the application state. A single place in the application that survives the redeploy and doesn’t do a proper cleanup is enough to sprout the leak. In a typical application there will be several such places, some of them almost impossible to fix due to the way third-party libraries are built. Leaking a classloader is therefore, quite common.

To examine this from a different perspective let’s return to the code example from our previous article. Breeze through it to quickly catch up.

Introducing the Leak

We will use the exact same Main class as before to show what a simple leak could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Main {
  private static IExample example1;
  private static IExample example2;
 
  public static void main(String[] args)  {
    example1 = ExampleFactory.newInstance().copy();
 
    while (true) {
      example2 = ExampleFactory.newInstance().copy();
 
      System.out.println("1) " +
        example1.message() + " = " + example1.plusPlus());
      System.out.println("2) " +
        example2.message() + " = " + example2.plusPlus());
      System.out.println();
 
      Thread.currentThread().sleep(3000);
    }
  }
}

The ExampleFactory class is also exactly the same, but here’s where things get leaky. Let’s introduce a new class called Leak and a corresponding interface ILeak:

1
2
3
4
5
6
7
8
9
10
interface ILeak {
}
 
public class Leak implements ILeak {
  private ILeak leak;
 
  public Leak(ILeak leak) {
    this.leak = leak;
  }
}

As you can see it’s not a terribly complicated class: it just forms a chain of objects, with each doing nothing more than holding a reference to the previous one. We will modify the Example class to include a reference to the Leak object and throw in a large array to take up memory (it represents a large cache). Let’s omit some methods shown in the previous article for brevity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Example implements IExample {
  private int counter;
  private ILeak leak;
 
  private static final long[] cache = new long[1000000];
 
  /* message(), counter(), plusPlus() impls */
 
  public ILeak leak() {
    return new Leak(leak);
  }
 
  public IExample copy(IExample example) {
    if (example != null) {
      counter = example.counter();
      leak = example.leak();
    }
    return this;
  }
}

The important things to note about Example class are:

  1. Example holds a reference to Leak, but Leak has no references to Example.
  2. When Example is copied (method copy() is called) a new Leak object is created holding a reference to the previous one.

If you try to run this code an OutOfMemoryError will be thrown after just a few iterations:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

With the right tools, we can look deeper and see how this happens.

Post Mortem

Since Java 5.0, we’ve been able to use the jmap command line tool included in the JDK distribution to dump the heap of a running application (or for that matter even extract the Java heap from a core dump). However, since our application is crashing we will need a feature that was introduced in Java 6.0: dumping the heap on OutOfMemoryError. To do that we only need to add -XX:+HeapDumpOnOutOfMemoryError to the JVM command line:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid37266.hprof ...
Heap dump file created [57715044 bytes in 1.707 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

After we have the heap dump we can analyze it. There are a number of tools (including jhat, a small web-based analyzer included with the JDK), but here we will use the more sophisticated Eclipse Memory Analyzer (EMA).

After loading the heap dump into the EMA we can look at the Dominator Tree analysis. It is a very useful analysis that will usually reliably identify the biggest memory consumers in the heap and what objects hold a reference to them. In our case it seems quite obvious that the Leak class is the one that consumes most of the heap:

classloader-dominator-tree.png

Now let’s run a search for all of the Leak objects and see what are they holding to. To do that we run a search List objects -> with outgoing references for “example.Leak”:

ema-search

The results include several Leak objects. Expanding the outgoing references we can see that each of them holds on to a separate instance of Example through a bunch of intermediate objects:

ema-analysis

You may notice that one of the intermediate objects is ExampleFactory$1, which refers to the anonymous subclass of URLClassLoader we created in the ExampleFactory class. In fact what is happening is exactly the situation we described in the beginning of the article:

  • Each Leak object is leaking. They are holding on to their classloaders
  • The classloaders are holding onto the Example class they have loaded:

classloader-leak

Conclusions

Though this example is slightly contrived, the main idea to take away is that it’s easy to leak a single object in Java. Each leak has the potential to leak the whole classloader if the application is redeployed or otherwise a new classloader is created. Since preventing such leaks is very challenging, it’s a better idea to use Eclipse Memory Analyzer and your understanding of classloaders to hunt them down after you get an OutOfMemoryError on redeploy.

This article addressed the following questions:

  • How does reloading a class cause the classloader to leak?
  • What are some consequences of leaking classloaders?
  • What tools can be used to troubleshoot these memory leaks?

Resources

Other Articles in the Reloading Java Classes Series

  • http://snowtide.com/ Chas Emerick

    The corollary to this is that, if one isn’t holding on to any object references from the old classloader regime, then leaks are strictly impossible. Correct, or no?

  • http://snowtide.com Chas Emerick

    The corollary to this is that, if one isn’t holding on to any object references from the old classloader regime, then leaks are strictly impossible. Correct, or no?

  • http://www.ekabanov.net/ Jevgeni Kabanov

    @Char
    The things that can hold the classloader are classes loaded with it, objects created from them and direct references to the classloader instance. If none of them are held, the classloader and its classes are garbage collected. However references can dangle from most surprising places, e.g. javax.security.auth.Policy saves a reference to the context classloader that you can’t remove no matter what you do.

  • http://www.ekabanov.net Jevgeni Kabanov

    @Char
    The things that can hold the classloader are classes loaded with it, objects created from them and direct references to the classloader instance. If none of them are held, the classloader and its classes are garbage collected. However references can dangle from most surprising places, e.g. javax.security.auth.Policy saves a reference to the context classloader that you can’t remove no matter what you do.

  • http://kohlerm.blogspot.com/ Markus Kohler

    Great blog! BTW the acronym we usually use it “MAT” :-)

  • http://kohlerm.blogspot.com Markus Kohler

    Great blog! BTW the acronym we usually use it “MAT” :-)

  • Khairul Ikhwan

    Great, i had this problem before and already solved it. Now, im working for new Java project for client and that project is already live on the production server which i cannot be remote from my office. I use tomcat6 to run it and i have file manager to upload or download any file from the server. When i made changes to the classes or library, i need to reload the application’s contact. Then i use tomcat application manager to stop and start that application. Then i face new problem that is OutOfMemoryError: PermGen space. This error is very bad as it will crashed the tomcat after a few minutes. When i use Java monitoring tools, i see that the PermGen space does not cleaned when i stop and start the context and will increase until it reach maximum. To overcome this, i use to set the MaxPermGenSpace in the Java_Opts. However, this only increase the memory and not sweep all the unused garbage.
    Maybe you can provide some tips on this problems.. :)

  • Khairul Ikhwan

    Great, i had this problem before and already solved it. Now, im working for new Java project for client and that project is already live on the production server which i cannot be remote from my office. I use tomcat6 to run it and i have file manager to upload or download any file from the server. When i made changes to the classes or library, i need to reload the application’s contact. Then i use tomcat application manager to stop and start that application. Then i face new problem that is OutOfMemoryError: PermGen space. This error is very bad as it will crashed the tomcat after a few minutes. When i use Java monitoring tools, i see that the PermGen space does not cleaned when i stop and start the context and will increase until it reach maximum. To overcome this, i use to set the MaxPermGenSpace in the Java_Opts. However, this only increase the memory and not sweep all the unused garbage.
    Maybe you can provide some tips on this problems.. :)

  • http://www.ekabanov.net/ Jevgeni Kabanov

    @Markus
    Thanks! MAT doesn’t make sense anymore :)

  • http://www.ekabanov.net Jevgeni Kabanov

    @Markus
    Thanks! MAT doesn’t make sense anymore :)

  • Sakuraba

    That and nothing else is the biggest issue of Java and it terrifies me that for at least a decade nothing has been achieved or at least been started by Sun to finally get rid of this mess.

  • Sakuraba

    That and nothing else is the biggest issue of Java and it terrifies me that for at least a decade nothing has been achieved or at least been started by Sun to finally get rid of this mess.

  • http://night-fairy-tales.blogspot.com/ SMiGL

    Great, Thanks!

  • http://night-fairy-tales.blogspot.com/ SMiGL

    Great, Thanks!

  • Ahriman

    Jevgeni,

    Can i make translation of this post and publish it to http://habrahabr.ru?
    Please answer ahriman@tpu.ru

  • http://zeroturnaround.com Jevgeni Kabanov

    You mean the whole series or this particular article?

  • Ahriman

    Sure.

  • Ahriman

    Jevgeni,
    Sorry, “sure” should be “sure the whole series”.

  • Valeriy Soglaev

    Евгений, я могу сделать перевод этого поста и опубликовать его на http://eProfil.net ?

  • http://www.facebook.com/ekabanov Jevgeni Kabanov

    Пожалуйста.

  • http://talonx.pip.verisignlabs.com/ Hrish

    On the contrary, a lot had been achieved by the erswhilte Sun in the form of optimizing the JVM and building some great tools to debug such issues.

  • bob

    Thread.currentThread().sleep(3000);

    should be written as
    Thread.sleep(3000);since sleep is a static method

  • Grudje

    Mine never crashes. I increased the cache to 50 million elements and as I type it has got to over 1000 iterations (I reduced the wait time !!). If I increase it to 100 million elements it crashes immediately with out of memory error.

  • Grudje

    I tried the downloaded version and that works although I did have to replace ExampleFactory.getClassPath() with the one that gets the real classpath. And I set the cache size at 10 million and it failed after 16 iterations.

  • Grudje

    OK eventually it did crash, after about 1500 iterations. Not sure what the difference was.