Blog

One common comment we hear when talking about JRebel is that class updates should be implemented in the standard JVM (see feature comparison and behind-the-scenes notes to find out more about JRebel and HotSwap). However even if Oracle or IBM would announce tomorrow that they implemented the support for full schema change HotSwap in the next version of the JVM, JRebel would still be a worthwhile investment. Why?

1. Adding new classes

HotSwap protocol works by specifying the class name and the updated bytecode. It works for existing classes because there’s a relatively unique correspondence between the class name and its content. HotSwap ignores the fact that each class is loaded in a specific class loader. However to add a new class you need to choose that class loader, which HotSwap doesn’t support.

In exploded deployment (where WARs, EARs and JARs are deployed unpacked as directories) new classes can be loaded directly from the file system. But in packaged deployment (WARs, EARs and JARs deployed as archives) adding new classes is a challenge. Even adding the new class to the archive will not do the trick, as the archives are commonly unpacked when deployed and updating the archive either causes redeployment or no change whatsoever.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding class loader for a specified application. This would need to be implemented by the application server.
• Extensions to the HotSwap protocol to allow sending new classes for a specified class loader, OR a different API/protocol to allow remote interaction with the IDE. This would need to be implemented in the JVM.
• each IDE to be aware of what application the new classes belong to. This would need to be supported by the IDE.

JRebel solves this by

• Extending the application server class loaders to search for classes from JRebel sources.
• Using the rebel.xml configuration file to match each specific class loader to the classes in the IDE.

2. Resources

The HotSwap protocol doesn’t make any mention of the resources packaged along with classes. Java applications include quite a few of these (XMLs, property files, graphic files, etc). If you use packaged deployment these resources will not be updated until you make a full build and redeploy.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding class loader for a specified application. This would need to be implemented by the application server.
• Extensions to the ClassLoader API to allow adding external sources for finding resources. This would need to be implemented by the application server.
• Extensions to the HotSwap protocol to allow sending resources for a specified class loader or a different API/protocol to allow remote interaction with the IDE. This would need to be implemented in the JVM. A big problem with this approach is that resource names are often not unique and several distinct resources with the same name in the same application are quite common.
• The IDE to be aware of which application the resources belong to. This would need to be supported by the IDE. Possibly you’d need to know even more about resources to solve the problem described in the previous point.

JRebel solves this by

• Extending the application server class loaders to search for resources from JRebel sources.
• Using the rebel.xml configuration file to match specific class loader to the resources in the IDE. JRebel solves the problem with multiple resources by giving the user control via rebel.xml to affect the order in which those resources are returned to the application.

3. Web resources

Besides the class loader resources, that are looked up via the ClassLoader API, every web application has a wealth of web resources like HTML, JSP, CSS, JS, images and so on. Unlike the class loader resources these are available in two ways:

1. Served to the end user via the HTTP protocol.
2. Read by the application using the ServletContext API.

If you use packaged deployment, then to update these resources you would need to do a full build and redeploy. HotSwap isn’t aware of these resources (and probably shouldn’t be), since ClassLoaders do not participate in any way in finding these resources.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding ServletContext or similar API for a specified application. This would need to be implemented by the application server.
• A remote API for updating/injecting web resources into a specified ServletContext. These resources must also be served via HTTP when requested. This would need to be implemented by the application server. Luckily, unlike the class loader resources, web resources are uniquely identified by their name.
• The IDE to be aware of which application the web resources belong to. This would need to be supported by the IDE.

JRebel solves this by

• Extending the application server ServletContext implementation and HTTP server implementations to search for web resources from JRebel sources.
• Using the rebel.xml configuration file to match a specific web application to the web resources in the IDE.

4. Caches

So let’s say the powers that be in the Java world have teamed up and implemented everything we described so far. You start your application, add a new method to the class, open your browser and stare at a juicy MethodNotFoundError. As it turns out, Java servers and frameworks tend to aggressively cache:

• Class structure information. This means that the servers/frameworks won’t be aware of the new methods and will fail. A good example is JSP EL and Scriptlets, which will fail in most implementations when trying to access a new method.
• Resources. Here we have both positive caches (the updated content is ignored) and negative caches (the new resource isn’t visible). Almost every server and some frameworks will include both of these caches.
• Configuration. Many resources (and class annotations) are only scanned once, and all changes are ignored afterwards. E.g. Spring XML configuration or Annotations.

This means that even after both JVM and the application server APIs are updated to support application updates, you still need to re-engineer a lot of Java code to accept the fact that classes and resources can change.

JRebel solves this by

• Purging the class structure information using listeners called when a class is updated and a lot of bytecode instrumentation to change the current framework implementations.
• Integrating with servers and frameworks to intelligently disable or alter the positive and negative caches. JRebel introduces its own update-aware cache to speed up resource lookup and delivery.
• Deeply integrating with frameworks to allow updates to the configuration. E.g. for Spring, JRebel causes the XMLs to be reread and the classpath to be rescanned.

5. Managed Components (EJBs, Spring beans,…)

Finally, when all is said and done, we still have the framework- or server-managed components like EJBs. These components often have:

• Some kind of state associated with an instance or an instance pool.
• Layers of proxies implementing aspects of functionality.
• Dependency injection.

These features mean that updating the class schema is nowhere near enough. When a class is updated:

• State may need to be updated to correspond to changes in the configuration. E.g. another method should now be executed inside a transaction.
• Proxies may need to be reconstructed or updated to apply to the added methods. Otherwise you might be looking at the MethodNotFoundError again.
• New dependencies must be injected. Otherwise NullPointerExceptions will abound.

Implementing this in current servers and frameworks will mean a substantial re-engineering across the codebase. JRebel solves this by re-implementing some server/framework features to allow for such updates to take place.

Conclusions

Although it could be possible to implement all of JRebel’s features in the Java ecosystem, it would take much more than just an update to the JVM. You’d need to re-engineer standards, introduce new APIs, implement them across app servers and then re-engineer much of the existing server/framework code base to take the possibility of updates into account. JRebel not only allows updates to class schema to take place, but also solves all those problems so that you can just update your code and see the change immediately. On average, that saves 2.5 minutes per redeploy — 5 times per hour of coding, and as Craig Pardey noticed, a whole bunch of context switching as well.

We are excited to announce another milestone on the way to the JRebel 3.0. This release includes the following new features:

• Advanced EJB Support. JRebel now includes plugins for JBoss 4.x, 5.x and Oracle Weblogic 8.x, 9.x and 10.x that support changing the Local/Remote EJB interface in EJB 1.x, 2.x and 3.x. It also supports on-the-fly dependency injection for @EJB annotation in JBoss. In the coming milestones we’ll add support for IBM WebSphere and more comprehensive support for the EJB 3.x features
• Startup time improvements. We significantly improved class/resource lookup overhead with caching and better server integration. Our nightly users are reporting up to 2x improvements in startup time.
• More integration. We added or improved plugins for Guice 2.x, JBoss Seam, Groovy, Wicket 1.4, Google App Engine, AspectJ, Struts 1 and Spring MVC. With every new plugin we are a step closer to our vision of seamless updates to your application, no matter what are you changing. Expect even more integration in the next JRebel release.

In addition to that there are numerous fixes and minor features that you can see in the changelog. And it includes the features introduced in 3.0 M1; in case you forgot, here they are:

• Support for adding static fields and changing enums. Previously when you added a static field to your class you’d see a discouraging warning in the console and get an exception when trying to access the field. Now JRebel will happily print “Reinitialized class” and your application will continue working as if nothing happened. To top it off you can also now change Java 1.5 enums in any way you like and they should just continue working. This feature is still a tad experimental, so please help us out to smooth it out!
• Full JSP <scriptlet> support. Now when you change your Java code (e.g. add a method) it can be immediately used in the Java code snippets in the JSP.
• 25%-30% less memory use. It is not uncommon to need a lot of PermGen heap with JRebel enabled. Now we optimized the memory use and will continue to drive it down in the upcoming milestones.

Download and enjoy!

In this article we’ll review how classes can be reloaded without dynamic class loaders. We will take a look at the JVM HotSwap class reloading support, Instrumentation API and ZeroTurnaround’s JRebel.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

HotSwap and Instrumentation

In 2002, Sun introduced a new experimental technology into the Java 1.4 JVM, called HotSwap. It was incorporated within the Debugger API, and allowed debuggers to update class bytecode in place, using the same class identity. This meant that all objects could refer to an updated class and execute new code when their methods were called, preventing the need to reload a container whenever class bytecode was changed. All modern IDEs (including Eclipse, IDEA and NetBeans) support it. As of Java 5 this functionality is also available directly to Java applications, through the Instrumentation API.

Unfortunately, this redefinition is limited only to changing method bodies — it cannot either add methods or fields or otherwise change anything else, except for the method bodies. This limits the usefulness of HotSwap, and it also suffers from other problems:

• The Java compiler will often create synthetic methods or fields even if you have just changed a method body (e.g. when you add a class literal, anonymous and inner classes, etc).
• Running in debug mode will often slow the application down or introduce other problems

This causes HotSwap to be used less than, perhaps, it should be.

Why is HotSwap limited to method bodies?

This question has been asked a lot during the almost 10 years since the introduction of HotSwap. One of the most voted for bugs for the JVM calls for supporting a whole array of changes, but so far it has not been implemented.

A disclaimer: I do not claim to be a JVM expert. I have a good general idea how the JVM is implemented and over the years I talked to a few (ex-)Sun engineers, but I haven’t verified everything I’m saying here against the source code. That said, I do have some ideas as to the reasons why this bug is still open (but if you know the reasons better, feel free to correct me).

The JVM is a heavily optimized piece of software, running on multiple platforms. Performance and stability are the highest priorities. To support them in different environments the Sun JVM features:

• Two heavily optimized Just-In-Time compilers (-client and -server)
• Several multi-generational garbage collectors

These features make evolving the class schema a considerable challenge. To understand why, we need to look a little closer as to what exactly is necessary to support adding methods and fields (and even more advanced, changing the inheritance hierarchy).

When loaded into the JVM, an object is represented by a structure in memory, occupying a continuous region of memory with a specific size (its fields plus metadata). In order to add a field, we would need to resize that structure, but since nearby regions may already be occupied, we would need to relocate the whole structure to a different region where there is enough free space to fit it in. Now, since we’re actually updating a class (and not just a single object) we would have to do this to every object of that class.

In itself this would not be hard to achieve — Java garbage collectors already relocate objects all the time. The problem is that the abstraction of one “heap” is just that, an abstraction. The actual layout of memory depends on the garbage collector that is currently active and, to be compatible with all of them, the relocation should probably be delegated to the active garbage collector. The JVM will also need to be suspended for the time of relocation, so doing GC at the same time makes sense.

Adding a method does not require updating the object structure, but it does require updating the class structure, which is also present on the heap. But consider this: the moment after a class has been loaded it is essentially is frozen forever. This enables the JIT to perform the main optimization that the JVM does — inlining. Most of the method calls in your application hot spots are eliminated and the code is copied to the calling method. A simple check is inserted to ensure that the target object is indeed what we think it is.

Here’s the punchline: the moment we can add methods to classes this “simple check” is not enough. We would need a considerably more complicated check that needs to ensure not only that no methods with the same name were added to the target class, but also to all it’s superclasses. Alternatively we could track all the inlined spots and their dependencies and deoptimize them when a class is updated. Either way it has a cost in either performance or complexity.

On top of that, consider that we’re talking about multiple platforms with varying memory models and instructions sets that probably require at least some specific handling and you get yourself an expensive problem with not much return on investment.

Introducing JRebel

In 2007, ZeroTurnaround announced the availability of a tool called JRebel (then JavaRebel) that could update classes without dynamic class loaders and with very few limitations. Unlike HotSwap, which is dependent on IDE integration, the tool works by monitoring the actual compiled .class files on disk and updating the classes whenever the files are updated. This means that you can use JRebel with a text editor and command-line compiler if so willing. Of course, it’s also integrated neatly into Eclipse, IntelliJ, and NetBeans. Unlike dynamic classloaders, JRebel preserves the identity and state of all existing objects and classes, allowing developers to continue using their application without delay.

How does this work?

For starters, JRebel works on a different level of abstraction than HotSwap. Whereas HotSwap works at the virtual machine level and is dependent on the inner workings of the JVM, JRebel makes use of two remarkable features of the JVM — abstract bytecode and classloaders. Classloaders allow JRebel to recognize the moment when a class is loaded, then translate the bytecode on-the-fly to create another layer of abstraction between the virtual machine and the executed code.

Others have used this features to enable profilers, performance monitoring, continuations, software transactional memory and even distributed heap. Combining bytecode abstraction with classloaders is a powerful combination, and can be used to implement a variety of features even more exotic than class reloading. As we examine the issue closer, we’ll see that the challenge is not just in reloading classes, but also doing so without a visible degradation in performance and compatibility.

As we reviewed in Reloading Java Classes 101 the problem in reloading classes is that once a class has been loaded it cannot be unloaded or changed; but we are free to load new classes as we please. To understand how we could theoretically reload classes, let’s take a look at dynamic languages on the Java platform. Specifically, let’s take a look at JRuby (we’ll simplify a lot, so don’t crucify anyone important).

Although JRuby features “classes”, at runtime each object is dynamic and new fields and methods can be added at any moment. This means that a JRuby object is not much more than a Map from method names to their implementations and from field names to their values. The implementations for those methods are contained in anonymously named classes that are generated when the method is encountered. If you add a method, all JRuby has to do is generate a new anonymous class that includes the body of that method. As each anonymous class has a unique name there are no issues loading it and as a result the application is updated on-the-fly.

Theoretically, since bytecode translation is usually used to modify the class bytecode, there is no reason why we can’t use the information in that class and just create as many classes as necessary to fulfill its function. We could then use the same transformation as JRuby and split all Java classes into a holder class and method body classes. Unfortunately, such an approach would be subject to (at least) the following problems:

• Perfomance. Such a setup would mean that each method invocation would be subject to indirection. We could optimize, but the application would be at least an order of magnitude slower. Memory use would also skyrocket, as so many classes are created.
• Java SDK classes. The classes in the Java SDK are considerably harder to process than the ones in the application or libraries. Also they often are implemented in native code and cannot be transformed in the “JRuby” way. However if we leave them as is, then we’ll cause numerous incompatibility errors, which are likely not possible to work around.
• Compatibility. Although Java is a static language it includes some dynamic features like reflection and dynamic proxies. If we apply the “JRuby” transformation none of those features will work unless we replace the Reflection API with our own classes, aware of the transformation.

Therefore, JRebel does not take such an approach. Instead it uses a much more complicated approach, based on advanced compilation techniques, that leaves us with one master class and several anonymous support classes backed by the JIT transformation runtime that allow modifications to take place without any visible degradation in performance or compatibility. It also

• Leaves as many method invocations intact as possible. This means that JRebel minimizes its performance overhead, making it lightweight.
• Avoids instrumenting the Java SDK except in a few places that are necessary to preserve compatibility.
• Tweaks the results of the Reflection API, so that we can correctly include the added/removed members in these results. This also means that the changes to Annotations are visible to the application.

Beyond Class Reloading – Archives

Reloading classes is something Java developers have complained about for a long time, but once we solved it, other problems turned up.

The Java EE standard was developed without much concern for development Turnaround (the time it takes between making a change to code and seeing the effects of that change in an application). It expects that all applications and their modules be packaged into archives (JARs, WARs and EARs), meaning that before you can update any file in your application, you need to update the archive — which is usually an expensive operation involving a build system like Ant or Maven. As we discussed in Reloading Java Classes 301 this can be minimized by using exploded development and incremental IDE builds, but for large application this is commonly not a viable option.

To solve this problem in JRebel 2.x we developed a way for the user to map archived applications and modules back to the workspace — our users create a rebel.xml configuration file in each application and module that tells JRebel where the source files can be found. JRebel integrates with the application server, and when a class or resource is updated it is read from the workspace instead of the archive.

This allows for instant updates of not just classes, but any kind of resources like HTML, XML, JSP, CSS, .properties and so on. Maven users don’t even need to create a rebel.xml file, since our Maven plugin will generate it automatically.

Beyond Class Reloading – Configurations and Metadata

En route to eliminating Turnaround, another issue becomes obvious: Nowadays, applications are not just classes and resources, they are wired together by extensive configuration and metadata. When that configuration changes it should be reflected in the running application. However it’s not enough to make the changes to the configuration files visible, the specific framework must reload it and reflect the changes in the application.

To support these kinds of changes in JRebel we developed an open source API that allows our team and third party contributers to make use of JRebel’s features and propagate changes in configuration to the framework, using framework-specific plugins. E.g. we support adding beans and dependencies in Spring on-the-fly as well as a wide variety of changes in other frameworks.

Conclusions

This article sums up the methods to reload Java classes without dynamic class loaders. We also discuss the reasons for HotSwap’s limitations, how JRebel works behind the scenes and the problems that arise when class reloading is solved.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

In this article we’ll review how dynamic classloaders are used in real servers, containers and frameworks to reload Java classes and applications.  We’ll also touch on how to get faster reloads and redeploys by using them in optimal ways.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

(more…)

We’re glad to announce the JRebel 2.2.1 release. It is a maintenance release incorporating all the bugfixes that have made since the 2.2 release. You can see the details from the full changelog.

Pick up the new version at our download page.

It is our great pleasure to announce JRebel 2.2, the “Easy Peasy” release. In this release we have focused heavily on ease of installation, configuration and use. The main new feature is the semi-automatic installer and configuration wizard, that makes installing JRebel and configuring your application a snap. We have also included a configuration utility that supplements all those funky system properties with a centralized GUI configuration. For those who prefers the Zen of the Command Line we have compiled a comprehensive reference manual about all things JRebel included in the distribution.

We also invested heavily into making the JRebel IDE integration as seamless as possible. The new NetBeans plugin now supports debugging when the JRebel agent is enabled. The numerous updates to Eclipse and IntelliJ IDEA plugins allow you to run the application with JRebel effortlessly from inside the IDE.

Take a look at the full changelog, download now or check out the screenshots:

Next week we plan to release the 2.2 version of JRebel. The two main features of this release are:

• The new semi-automatic installer and configuration wizard, that make installing and configuring JRebel a snap.
• New or updated releases of IDE plugins for Eclipse, IntelliJ and NetBeans (both 6.5 and 6.7).

While we’re finishing the polish on the release itself, we have already uploaded the IDE plugins to their respective repositories. Just hit “Update” and you can start using them right away. We have also uploaded a preview version of the Installer-enabled JRebel distribution, which you can now get from Downloads.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

A video presentation “Do you really get class loaders” by Jevgeni Kabanov

From ClassLoaders to Classes

If you have programmed in Java for some time you know that memory leaks do happen. Usually it’s the case of a collection somewhere with references to objects (e.g. listeners) that should have been cleared, but never were. Classloaders are a very special case of this, and unfortunately, with the current state of the Java platform, these leaks are both inevitable and costly: routinely causing OutOfMemoryError’s in production applications after just a few redeploys.

Let’s get started. Recalling RJC101: to reload a class we threw away the old classloader and created a new one, copying the object graph as best we could:

Every object had a reference to its class, which in turn had a reference to its classloader. However we didn’t mention that every classloader in turn has a reference to each of the classes it has loaded, each of which holds static fields defined in the class:

This means that

1. If a classloader is leaked it will hold on to all its classes and all their static fields. Static fields commonly hold caches, singleton objects, and various configuration and application states. Even if your application doesn’t have any large static caches, it doesn’t mean that the framework you use doesn’t hold them for you (e.g. Log4J is a common culprit as it’s often put in the server classpath). This explains why leaking a classloader can be so expensive.
2. To leak a classloader it’s enough to leave a reference to any object, created from a class, loaded by that classloader. Even if that object seems completely harmless (e.g. doesn’t have a single field), it will still hold on to its classloader and all the application state. A single place in the application that survives the redeploy and doesn’t do a proper cleanup is enough to sprout the leak. In a typical application there will be several such places, some of them almost impossible to fix due to the way third-party libraries are built. Leaking a classloader is therefore, quite common.

To examine this from a different perspective let’s return to the code example from our previous article. Breeze through it to quickly catch up.

Introducing the Leak

We will use the exact same Main class as before to show what a simple leak could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class Main {
  private static IExample example1;
  private static IExample example2;
 
  public static void main(String[] args)  {
    example1 = ExampleFactory.newInstance().copy();
 
    while (true) {
      example2 = ExampleFactory.newInstance().copy();
 
      System.out.println("1) " +
        example1.message() + " = " + example1.plusPlus());
      System.out.println("2) " +
        example2.message() + " = " + example2.plusPlus());
      System.out.println();
 
      Thread.currentThread().sleep(3000);
    }
  }
}

The ExampleFactory class is also exactly the same, but here’s where things get leaky. Let’s introduce a new class called Leak and a corresponding interface ILeak:

1
2
3
4
5
6
7
8
9
10

interface ILeak {
}
 
public class Leak implements ILeak {
  private ILeak leak;
 
  public Leak(ILeak leak) {
    this.leak = leak;
  }
}

As you can see it’s not a terribly complicated class: it just forms a chain of objects, with each doing nothing more than holding a reference to the previous one. We will modify the Example class to include a reference to the Leak object and throw in a large array to take up memory (it represents a large cache). Let’s omit some methods shown in the previous article for brevity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class Example implements IExample {
  private int counter;
  private ILeak leak;
 
  private static final long[] cache = new long[1000000];
 
  /* message(), counter(), plusPlus() impls */
 
  public ILeak leak() {
    return new Leak(leak);
  }
 
  public IExample copy(IExample example) {
    if (example != null) {
      counter = example.counter();
      leak = example.leak();
    }
    return this;
  }
}

The important things to note about Example class are:

1. Example holds a reference to Leak, but Leak has no references to Example.
2. When Example is copied (method copy() is called) a new Leak object is created holding a reference to the previous one.

If you try to run this code an OutOfMemoryError will be thrown after just a few iterations:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

With the right tools, we can look deeper and see how this happens.

Post Mortem

Since Java 5.0, we’ve been able to use the jmap command line tool included in the JDK distribution to dump the heap of a running application (or for that matter even extract the Java heap from a core dump). However, since our application is crashing we will need a feature that was introduced in Java 6.0: dumping the heap on OutOfMemoryError. To do that we only need to add -XX:+HeapDumpOnOutOfMemoryError to the JVM command line:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid37266.hprof ...
Heap dump file created [57715044 bytes in 1.707 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

After we have the heap dump we can analyze it. There are a number of tools (including jhat, a small web-based analyzer included with the JDK), but here we will use the more sophisticated Eclipse Memory Analyzer (EMA).

After loading the heap dump into the EMA we can look at the Dominator Tree analysis. It is a very useful analysis that will usually reliably identify the biggest memory consumers in the heap and what objects hold a reference to them. In our case it seems quite obvious that the Leak class is the one that consumes most of the heap:

Now let’s run a search for all of the Leak objects and see what are they holding to. To do that we run a search List objects -> with outgoing references for “example.Leak”:

The results include several Leak objects. Expanding the outgoing references we can see that each of them holds on to a separate instance of Example through a bunch of intermediate objects:

You may notice that one of the intermediate objects is ExampleFactory$1, which refers to the anonymous subclass of URLClassLoader we created in the ExampleFactory class. In fact what is happening is exactly the situation we described in the beginning of the article:

• Each Leak object is leaking. They are holding on to their classloaders
• The classloaders are holding onto the Example class they have loaded:

Conclusions

Though this example is slightly contrived, the main idea to take away is that it’s easy to leak a single object in Java. Each leak has the potential to leak the whole classloader if the application is redeployed or otherwise a new classloader is created. Since preventing such leaks is very challenging, it’s a better idea to use Eclipse Memory Analyzer and your understanding of classloaders to hunt them down after you get an OutOfMemoryError on redeploy.

This article addressed the following questions:

• How does reloading a class cause the classloader to leak?
• What are some consequences of leaking classloaders?
• What tools can be used to troubleshoot these memory leaks?

Resources

• How to fix the dreaded “java.lang.OutOfMemoryError: PermGen space” exception (classloader leaks)
• Download full source code

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

We’re glad to announce the JRebel 2.1.1 release. It is a maintenance release incorporating all the bugfixes that have accumulated during the past month or so. It also includes the new Log4J plugin, support for Jetty 7 and GlassFish v3 Preview (apparently Prelude and Preview differ a lot, go figure).

Changes include:

• Support for Jetty 7
• Preliminary support for GlassFish v3 Preview
• Log4J plugin now reloads changes to log configuration on-the-fly, contributed by Julien Richard.
• Fixed an issue causing the annotations on constructors to disappear after class reload.
• Fixed an issue with Google Web Toolkit client side classes in hosted mode
• Fixed an issue with FileNotFoundException thrown by JavaRebelResourceServlet
• Fixed an issue with ClassCastException when defining web services in web.xml
• Fixed several issues in the Wicket plugin.

You can pick up the new version on our download page, by choosing the standard download.

Welcome to Turnaround article series from ZeroTurnaround.

In this article we will review how to reload a Java class using a dynamic classloader. To get there we'll see how objects, classes and classloaders are tied to each other and the process required to make changes. We begin with a bird's eye view of the problem, explains the reloading process, and then proceed to a specific example to illustrate typical problems and solutions. Other articles in the series include:

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development -- Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

A video presentation "Do you really get class loaders" by Jevgeni Kabanov

A Bird's Eye View

The first thing to understand when talking about reloading Java code is the relation between classes and objects. All Java code is associated with methods contained in classes. Simplified, you can think of a class as a collection of methods, that receive "this" as the first argument. The class with all its methods is loaded into memory and receives a unique identity. In the Java API this identity is represented by an instance of java.lang.Class that you can access using the MyObject.class expression.

Every object created gets a reference to this identity accessible through the Object.getClass() method. When a method is called on an object, the JVM consults the class reference and calls the method of that particular class. That is, when you call mo.method() (where mo is an instance of MyObject), then the JVM will call mo.getClass().getDeclaredMethod("method").invoke(mo) (this is not what the JVM actually does, but the result is the same).

Every Class object is in turn associated with its classloader (MyObject.class.getClassLoader()). The main role of the class loader is to define a class scope -- where the class is visible and where it isn't. This scoping allows for classes with the same name to exist as long as they are loaded in different classloaders. It also allows loading a newer version of the class in a different classloader.

The main problem with code reloading in Java is that although you can load a new version of a class, it will get a completely different identity and the existing objects will keep referring the previous version of the class. So when a method is called on those objects it will execute the old version of the method.

Let's assume that we load a new version of the MyObject class. Let's refer to the old version as MyObject_1 and to the new one as MyObject_2. Let's also assume that MyObject.method() returns "1" in MyObject_1 and "2" in MyObject_2. Now if mo2 is an instance of MyObject_2:

• mo.getClass() != mo2.getClass()
• mo.getClass().getDeclaredMethod("method").invoke(mo)
!= mo2.getClass().getDeclaredMethod("method").invoke(mo2)
• mo.getClass().getDeclaredMethod("method").invoke(mo2) throws a ClassCastException, because the Class identities of mo and mo2 do no match.

This means that any useful solution must create a new instance of mo2 that is an exact copy of mo and replace all references to mo with it. To understand how hard it is, remember the last time you had to change your phone number. It's easy enough to change the number itself, but then you have to make sure that everyone you know will use the new number, which is quite a hassle. It's just as difficult with objects (in fact, it's actually impossible, unless you control the object creation yourself), and we're talking about many objects that you must update at the same time.

Down and Dirty

Let's see how this would look in code. Remember, what we're trying to do here is load a newer version of a class, in a different classloader. We'll use an Example class that looks like this:

public class Example implements IExample {
  private int counter;
  public String message() {
    return "Version 1";
  }
  public int plusPlus() {
    return counter++;
  }
  public int counter() {
    return counter;
  }
}

We'll use a main() method that will loop infinitely and print out the information from the Example class. We'll also need two instances of the Example class: example1 that is created once in the beginning and example2 that is recreated on every roll of the loop:

public class Main {
  private static IExample example1;
  private static IExample example2;
 
  public static void main(String[] args)  {
    example1 = ExampleFactory.newInstance();
 
    while (true) {
      example2 = ExampleFactory.newInstance();
 
      System.out.println("1) " +
        example1.message() + " = " + example1.plusPlus());
      System.out.println("2) " +
        example2.message() + " = " + example2.plusPlus());
      System.out.println();
 
      Thread.currentThread().sleep(3000);
    }
  }
}

IExample is an interface with all the methods from Example. This is necessary because we'll be loading Example in an isolated classloader, so Main cannot use it directly (otherwise we'd get a ClassCastException).

public interface IExample {
  String message();
  int plusPlus();
}

From this example, you might be surprised to see how easy it is to create a dynamic class loader. If we remove the exception handling it boils down to this:

public class ExampleFactory {
  public static IExample newInstance() {
    URLClassLoader tmp =
      new URLClassLoader(new URL[] {getClassPath()}) {
        public Class loadClass(String name) {
          if ("example.Example".equals(name))
            return findClass(name);
          return super.loadClass(name);
        }
      };
 
    return (IExample)
      tmp.loadClass("example.Example").newInstance();
  }
}

The method getClassPath() for the purposes of this example could return the hardcoded classpath. However, in the full source code (available in the Resources section below) you can see how we can use the ClassLoader.getResource() API to automate that.

Now let's run Main.main and see the output after waiting for a few loop rolls:

1) Version 1 = 3
2) Version 1 = 0

As expected, while the counter in the first instance is updated, the second stays at "0". If we change the Example.message() method to return "Version 2". The output will change as follows:

1) Version 1 = 4
2) Version 2 = 0

As we can see, the first instance continues incrementing the counter, but uses the old version of the class to print out the version. The second instance class was updated, however all of the state is lost.

To remedy this, let's try to reconstruct the state for the second instance. To do that we can just copy it from the previous iteration.

First we add a new copy() method to Example class (and corresponding interface method):

  public IExample copy(IExample example) {
    if (example != null)
      counter = example.counter();
    return this;
  }

Next we update the line in the Main.main() method that creates the second instance:

example2 = ExampleFactory.newInstance().copy(example2);

Now waiting for a few iterations yields:

1) Version 1 = 3
2) Version 1 = 3

And changing Example.message() method to return "Version 2" yields:

1) Version 1 = 4
2) Version 2 = 4

As you can see even though it's possible for the end user to see that the second instance is updated and all its state is preserved, it involves managing that state by hand. Unfortunately, there is no way in the Java API to just update the class of an existing object or even reliably copy its state, so we will always have to resort to complicated workarounds.

In subsequent articles we'll review how web containers, OSGi, Tapestry 5, Grails and others confront the problem of managing state when reloading classes, then we'll dig into how HotSwap, Dynamic Languages, and the Instrumentation API work, and go behind the scenes with JRebel as well.

Resources

• Internals of Java Class Loading
• Download full source code

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development -- Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

« Newer Entries | Older Entries »