Open the Black Box – Oracle JRockit, Sun Hotspot and IBM J9 JVM Tools

 Are you aware of the different tools available in the JVM you use to help troubleshoot and look under the hood?  Over the past couple of weeks several of my customers have been involved in several WebLogic support escalations with different JVM vendors.  It was an opportunity for me to learn a about the different tools – some unique to each JVM – and I want to document what I’ve found thus far.  I am by no means an expert in this area, but based on what I’ve seen there is not a high-level of awareness about the tools freely available.  I think you’ll find that there is a lot of innovation in JRockit Mission Control, and VisualVM is also a promising tool that you should get familiar with.  I purposely am not covering 3rd party commercial products as I am not familiar with them and the list is long.

Dip Your Toe In The Water With JConsole or VisualVM

Sun has a detailed article about common symptoms and JVM tools to use for each, which I highly recommend.  But if the JVM is mostly a black-box to you, then trying to analyze some of the low-level detail that comes out of the JVM tools can be very intimidating at first and you can start by keeping it simple.  Two important questions to ask for JVM health are:

1) Is the JVM on a sustainable path?

2) What is the JVM doing?

The easiest way to get a quick answer to these questions that works across all JVM vendors and modern releases is JConsole because it ships with the JDK, so you know it’s there without a separate download.  The JDK documentation shows you how to use it on either a local or remote JVM.  Starting with JDK 1.6 update 7 there is a better tool called visualvm that ships with the JDK, but it can also be downloaded and used (with JDK 1.6) to monitor 1.4.2 and later JDKs.  I’m going to show visualvm because it’s simply newer and better.

VisualVM Monitor Tab

This snapshot is a stable and sustainable JVM (WebLogic Server on a JRockitJDK) where trend lines are relatively flat.  The memory utilization is low and returns to normal levels after each Garbage Collection.  The threads are not constantly increasing, and the CPU utilization is very low.  If you saw either the memory or threads going up constantly or saw extremely high CPU utilization, it would be worth investigating further.  Trend lines that slope up and to the right are signs of an upcoming problem.

Visual VM Threads Tab

The Threads tab shows a color coded timeline for all the threads by state and also has sub-tabs with views to sort by table columns and show extra detail.  You can also generate a full thread dump from this tab, which will open the current execution stack for each thread in the JVM in a new tab.  This reminds me a lot of the Mission Control Latency Analyzer.

More on WebLogic Threads

Support often asks for several thread dumps taken at 10-15 second intervals when Service Requests are filed.  By looking at the current point of execution of each thread over a period of time, you can identify whether some threads are stuck on something that should be fast, identify resource contention, etc.

If you’re using WebLogic Server, you do not necessarily have to worry about how each JVM has different ways to create a thread dump and potentially even different output formats.  Simply use the WLST threadDump() command regardless of which JVM or platform you are on and it should work the same to produce text output.  Max actually has a good example of how to do this on a periodic basis.

The 2009 book Professional Oracle WebLogic Server has excellent information about how to read WLS thread dumps starting on page 686.  But for those of you that want Cliff Notes, STUCK threads are bad.  A thread that has the name STUCK in it’s name has been identified as not being returned to the thread pool for a configured amount of time, which defaults to 10 minutes.

Digging Deeper with Specialized JDK Tools

Thread Dump Analysis

If you want to use the JVM tools that help decipher thread dumps, then you may need another technique as the format of the thread dump varies depending on the vendor.  These can help you find lock contention, bogged down resources, and other nasty problems.

Oracle JRockit – There is not a dedicated tool for Thread Dumps in JRockit as the format is human readable.  The JRockit documentation has a section describing thread dump analysis.  JRockit Mission Control also has some thread analysis both the live Console tool and the JRockit Runtime Analyzer recordings files.  For an entire mapping of Sun JDK tools to JRockit tools, look at this page in the docs.

Sun Hotspot – the jdk\bin\jstack command produces output suitable for Thread Dump Analyzer, a project on java.net that has many useful features for analyzing multiple thread dumps.

IBM J9 – The IBM JDK does not use a human readable format for it’s core files, so it’s especially important to know about the IBM Thread and Monitor Dump Analyzer for Java hosted on AlphaWorks.  The author published an article in JDJ covering how it works.  It’s a nice tool.

Heap Analysis

Oracle JRockit Mission Control has both the Memory Leak Detector and the JRockit Runtime Analyzer that excellent and easy to use visual tools.  On the command line, JRCMD also has a heap_diagnostics command that produces a human readable text file.  Paul Done has written about it recently.

Sun Hotspot – At the lowest level, there is a JVM tool called Java Heap Analysis Tool or jhat that ships with the JDK.  Additionally VisualVM also includes heap dump analysis features.  Eclipse Memory Analysis Tool (MAT) also supports both Sun and IBM JVM binary heaps as has a really nice report for finding Leak Suspects that doesn’t require advanced JVM knowledge to use.

IBM J9 – There are several tools available including the IBM Support Assistant (ISA) Plugins, the original is the Memory Dump Diagnostic for Java.  There are several beta plugins for ISA that offer more functionality.  Eclipse MAT now supports the IBM JDK also.  The oldest tool to my knowledge is the AlphaWorks project Heap Analyzer.

Garbage Collection Analysis

GCViewer claims some support for all 3 of the vendors verbose logging formats.  It is not clear whether all new JDK releases are forward compatible.

Oracle JRockit – Instead of the Sun -verbose:gc option, JRockit has an -Xverbose:memory command.  Additionally, the JRockit Runtime Analyzer files have detailed GC information presented in an easy to understand way.

Sun Hotspot – -verbose:gc on the command line and a visual tool named appropriately visualgc.

IBM J9 – AlphaWorks has a tool called IBM Pattern Modeling and Analysis Tool for Java Garbage Collector that parses the IBM JDK verbose gc output and makes very nice graphs.

Summary

Overall I really like JRockit Mission Control and use it most often.  It ships with the JRockit JDK so there is no separate download.  It integrates with Eclipse (including OEPE) and links me directly to the line of code in JDT.  It has a helpful network auto discovery protocol built-in that don’t require me to install remote daemons on other machines to find remote JVMs.  Perhaps most importantly to me, I have the most experience with it, find it the easiest to use and most consolidated.  Truthfully, I haven’t given VisualVM enough screen-time yet for a fair comparison and there is an Eclipse launcher for it that I have yet to try.  I also have to give IBM credit for really having a nice vision with the IBM Support Assistant even though I don’t think it’s capabilities or user experience quite match what Oracle and Sun have yet strictly from a JDK tooling perspective.  I’m also not sure it will ever be shipped with the IBM JDK.  IBM ISA is intended for a broader-than-java product support perspective and has a larger scope than just the JDK.  If you have any other JDK vendor tools that I should have covered but omitted please let me know.