If a VM process appears to be looping, the first step is to try to get a thread dump. If a thread dump can be obtained, it will often be clear which thread is looping. If the looping thread can be identified, then the trace stack in the thread dump can provide direction on where (and maybe why) the thread is looping.
If the application console (standard input/output) is available, then press the Control+\ key combination (on Oracle Solaris or Linux) or the Control+Break key combination (on Windows) to cause the HotSpot VM to print a thread dump, including thread state. On Oracle Solaris and Linux operating systems the thread dump can also be obtained by sending a SIGQUIT to the process (command kill -QUIT pid). In this case the thread dump is printed to the standard output of the target process. The output might be directed to a file, depending on how the process was started.
If the Java process is started with the -XX:+PrintClassHistogram command-line option, then the Control+Break handler will produce a heap histogram.
If a thread dump can be obtained, then a good place to start is the thread stacks of the threads that are in the RUNNABLE state. See Thread Dump for information on the format of the thread dump, as well as a table of the possible thread states in the thread dump. In some cases it might be necessary to get a sequence of thread dumps in order to determine which threads appear to be continuously busy.
If the application console is not available (for example, the process is running in the background, or the VM output is directed to an unknown location), then the jstack utility can be used to obtain the stack thread. Use the jstack -F pid command to force a stack dump of the looping process. See The jstack Utility for information on the output of this utility. The jstack utility should also be used if the thread dump does not provide any evidence that a Java thread is looping.
When reviewing the output of the jstack utility, focus initially on the threads that are in the RUNNABLE state. This is the most likely state for threads that are busy and possibly looping. It might be necessary to execute jstack a number of times to get a more complete picture of which threads are looping. If a thread appears to be always in the RUNNABLE state, then the -m option can be used to print the native frames and provide a further hint on what the thread is doing. If a thread appears to be looping continuously while in the RUNNABLE state, this situation can indicate a potential HotSpot VM bug that needs further investigation.
If the VM does not respond to Control+\, this could indicate a VM bug rather than an issue with application or library code. In this case use jstack with the -m option (in addition to the -F option) to get a thread stack for all threads. The output will include the thread stacks for VM internal threads. In this stack trace, identify threads that do not appear to be waiting. For example, on Oracle Solaris operating system you identify the threads that are not in functions such as __lwp_cond_wait, __lwp_park, ___pollsys, or other blocking functions. If it appears that the looping is caused by a VM bug, then collect as much data as possible and submit a bug report. See Submit a Bug Report for more details on data collection.
