Top 5 Java Profilers Revealed: Real world data with VisualVM, JProfiler, Java Mission Control, YourKit and Custom tooling.
Earlier this year, we released our annual Developer Productivity report. It was based off a performance survey which collected responses from over 1500 developers, testers, architects, and many more interestingly named job titles. The great news is we gave $0.50 for every completed survey to a great charity called Dogs for Good (formerly Dogs for the Disabled), which provide assistance dogs to disabled children.
As with all problems which have many solutions, it can often be tricky to determine which is the best solution for you. What works for other people may not work for you. When it comes to profilers, there are many on the market, and it’s often tough to determine what the differences are and which is best for your application.
This blog post looks at some of the report insights and answers the following questions:
- Which tools find the highest number of bugs?
- Should I be using more than one tool?
Let’s look at the results!
Almost half (46.6%) of survey respondents use multiple profilers
The wily among you will have noticed that the first image in this blog contains percentages which do not add up to 100%. The reason for this isn’t basic math incompetence. No, not this time. This time it’s because the question allowed for multiple choice answers, in case people tended to use more than one profiling tool on their applications. It turns out that it was very common for teams to use multiple tools as each tool tends to be better in certain phases or the application lifecycle, or perhaps preferable when chasing down a particular type of performance bug.
Wanna find bugs? Roll your own!
Most performance bugs were found by custom in house tools. This might be because those who write the custom tools work with very particular applications and have the specific application knowledge to expose certain bugs that other tools may not. We might also be able to infer that if a team were able to write their own in house performance tools, they are likely to be closer to the nirvana that is a performance expert. As such they would be able to find more performance bugs than others anyway.
Tools which did very well include JProfiler, XRebel, NetBeans profiler, JMC and JProbe. From a personal point of view, it was really exciting to see XRebel, a relatively new profiler on the block finding some of the most bugs out of any tool. This may well be because it is designed to be used in development, a place where it’s easier to find and eliminate bugs.
Can you really blame your tools?
Well, from the graph below, you might be able to blame not having enough tools, as there’s a definite trend showing that you can find more performance issues if you use multiple tools. This helps to show there is a right tool for the right job. The job however may vary, for instance, you might be trying to find and fix a specific types of IO issues or code bottleneck. You might have a prefered tool for each of those tasks. You might want to use JMC in production, but XRebel in development as each tool is better suited to different environments.
The Top 5 Java Profilers!
As you have seen from the first image in the blog, the top five Java profilers, by usage, in our industry, from our survey results are: VisualVM, JProfiler, Custom tooling, Java Mission Control and YourKit. Let’s take a look at them in a little more depth. Oh we’ll also give an honourable mention to XRebel, the lightweight development time profiler taking over 3% of the votes, which wasn’t even a year old at the time the survey was released! We’ll look through them in reverse order of usage for increased tension and suspense, even though you already know which is top!
Your Kit (Used by 15.0% of respondents)
The YourKit profiler, is one of the most established leaders among Java profilers. As a mature and a versatile profiler, YourKit can do both CPU and memory profiling for you, with integrations across major Java application servers, JDBC and other frameworks for high-level performance analysis like finding synchronization issues and excessive database accesses. YourKit can run in both sampling and tracing profiling modes and the mixed approach helps it make the most from both worlds: the precision of tracing the actual code execution, while being able to precisely control the profiling overhead.
The memory profiling in YourKit can detect memory leaks and trace the excessive objects back to the GC roots to show you why the objects are not being collected. A memory snapshot comparison and automatic memory snapshot generation when memory is low can help further analyze your application’s heap. With the unique on-demand profiling, you can run the profiled application with approximately zero overhead, activating actual profiling only when it is needed. When the profiler’s capabilities are needed you can always turn the YourKit profiler on and you can precisely control the overhead that you’re able to tolerate.
YourKit, used by 15% of survey respondents didn’t fare too well on how many performance bugs typically found per release, but it wasn’t too far behind with 4.2 issues on average.
Java Mission Control (Used by 17.1% of respondents)
Java Mission Control is a Java performance monitoring tool by Oracle which has been shipped with the JDK since Java version 7 update 40. It encompases two JMX Console and Java Flight Recorder. Java Mission Control works by interacting with a JMX agent in a JVM which has an MBean server that integrates with the built in VM and app instrumentation running the in the JVM. That’s a key piece as it really does lower the usage cost of the tool as it’s using pre-existing hooks. Oracle state this is normally well below a 1% performance cost.
Flight recorder, with it’s historical view, provides the ability to see trends in your JVM. This gives you the data needed to find memory leaks, latency issues around thread waits, locking issues and more. The most unique feature Java Mission Control brings to the table is that it’s shipped with the Oracle JDK. There’s nothing that you need to install or attach to your existing VM to get it working. Start up a terminal go to your JDK bin directory and just type
Given Java Mission Control is packaged and shipped with the Oracle JDK, it’s surprising only 17.1% of respondents use it. I would expect the biggest issue around Java Mission Control is awareness. Even today when I ask how many people even know about it less than half of a room at a conference room will put their hands up. The tool fared reasonably well in the survey, finding an average of 5 performance issues per release.
Custom in-house tools (Used by 20.6% of respondents)
If you wanted to write custom tools to test your application, you need to be able to trust their results. In order to trust your tool’s results, you have to be smart enough to write the tool, taking account of all performance intricacies which existing profilers have spent years phasing out. Often this is more scenario based testing that writing an actual tool, but 1 in 5 people use their own tooling, which finds more bugs than other profilers.
JProfiler (Used by 25.7% of respondents)
JProfiler is a comprehensive profiler for Java SE and Java EE applications with plugins for all major IDEs which provides enhanced analysis of the collected profile data. As with any profiler, the CPU profiling is perhaps the most important and useful thing you can get from JProfiler. Additionally, JProfiler can show a call graph view, where the methods are represented by colored rectangles that provide instant visual feedback about where the slow code resides in the method call chains, making bottlenecks easier to find. Memory profiling with JProfiler can also be tuned to get more or less detail, for more data or to help reducing the performance overhead. It can collect, analyze and render snapshots of the heap created with HPROF.
From our survey we can see JProfiler is in use by 1 in 4 of our respondents and is the second most heavily used profiler. It also does particularly well when we look at how many bugs each profiler finds in a release. In fact it finishes second, only to custom in-house tooling, so is the highest placing third-party profiler in that category.
VisualVM (Used by 46.5% of respondents)
JVisualVM is an All-in-One Java Troubleshooting tool that takes advantage of several command line tools that are bundled with your JDK distribution and presents the information about your Java process performance in a nice UI. Besides being a lightweight monitoring tool, it offers application developers the ability to properly profile their code execution, collect and browse thread dumps and heap dumps, while gathering various statistics about the internals of the JVM.
One of the most under-appreciated features is the ability to monitor MXBeans, management components that can show you thread pools usage, memory pools statistics, details about the frequency of garbage collection and so on. Typical memory problems like analyzing a heap dump to see what objects take the most of your JVM process memory. While other profilers might offer a more advanced approach to application profiling, VisualVM is the most appropriate first tier tool to analyze the performance of your code. Given that it’s bundled with your JDK distribution and is literally just one click away, it’s an invaluable tool! No wonder almost one in two developers is using it to either fully profile the application or in conjunction with other profilers to get a good initial impression using VisualVM and perhaps digging deeper with something heavier.
While VisualVM did significantly turn out to be the most popular profiler among our respondents, by usage, the tool struggled more on the number of performance bugs it found. In fact it was turned out to be the tool which respondents said found fewest bugs in their environment. However as you might expect to use it in addition to other profilers, this may not be as big of an issue as it sounds.
XRebel – Honourable mention!
Because XRebel was only a year old when it exceeded 3% in our survey, oh and because it was the second highest profiler for finding the most bugs, oh and because we created it, we’ll be giving XRebel and honourable mention! :o)
XRebel by ZeroTurnaround is a performance tool for Java that is intended for use in a development environment setting. It’s a javaagent that instruments Java web-applications and automatically injects a widget like reporting console, embedded in your application view. The widget shows the application performance data. The main benefit comes from the fact that the developer can fix nastiest performance issues without even committing the poor code into the build.
XRebel gathers and nicely presents the time spent serving every request, broken down into relevant method calls. This application trace contains relative information about both self-time and total time of the methods together with an intelligent way of presenting the information that makes obvious which methods affect the performance the most. By only showing relevant method information, this approach scales with your applications
Additionally, XRebel has a tight integration with database drivers and common HTTP querying solutions, and can gather and display the database activity originated in the application and requests to third-party web-services. Database access and HTTP calls are the most common reasons for poor performance and XRebel shines at presenting the developer with accessible data about them. This makes avoiding certain performance issues easy.
XRebel finds excessive database access, easily uncovers the N+1 queries problems while still in development. It understands which objects clog the memory in the HTTP session as they happen during development testing. Also, it allows developers to instantly see all exceptions happening in the application, even if they are not propagated to the UI properly, aka, the hidden exception!
NB! If you’d like to use XRebel in continuous integration setting, check out XRebel Hub, it does exactly that!
This was quite a long post, so let’s summarise it in a few sentences.
- People who performance test their application using multiple profilers tend to find more performance issues than just using one.
- Visual VM is the most commonly used Java profiler.
- The Java profiling market is fairly well spread across a number of different tools.
- Custom tools fared very well when respondents reported how many performance bugs they found using them.
- JProfiler, XRebel and NetBeans profiler find the most performance bugs among our respondents.
I hope you found this post useful! If you did and want more information, be sure to check out the full report on one of the button links above. I’ll post a link to it here too so you have no excuse not to look through it! I’d love to hear your thoughts, so why not comment below on your favourite profiler or which combination you feel works best for your environment, or hit us up on twitter at @ZeroTurnaround.