kvg February 2016

JVM crashes when attempting to run multiple invocations on a high-memory, many CPU server

I've written some Java-based, single-threaded, genomics software that runs on one sample at a time, and I have hundreds of samples to process. I have access to a machine with 64 CPUs and a terabyte of RAM, and there are no other users on the system. The maximum heap size requested for each invocation is 8 gigabytes. I expect I should be able to invoke 30 instances of my code simultaneously (assuming two threads - my main thread and a GC thread?). I'm only attempting to dispatch 20 at a time (using a makefile and the -j20 argument). In practice, however, only 5 run. The rest fail with the message:

# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.

Suspicious that this had more to do with the simultaneous invocations than actual resource limitations, I implemented a random several-second delay in my execution loop for each invocation of my program. That gets me up to 10 programs running simultaneously, rather than 5, with the same failure message.

Questions:

  1. Why does attempting to invoke dozens of instances of the JVM simultaneously fail this way, despite doing so on a system that clearly has the resources available?

  2. Why does my hack implementing the dispatch delay clear up some of the problem?

  3. What's a better way to get all 20 instances running simultaneously?

Answers


Andreas February 2016

This is not really an answer, but a guide to help you research.

Create a small dummy program, e.g.

public final class Test {
    public static void main(String[] args) throws Exception {
        String runtimeName = ManagementFactory.getRuntimeMXBean().getName();
        System.out.println(runtimeName + " Start");
        Thread.sleep(10_000L); // 10 secs
        System.out.println(runtimeName + " End");
    }
}

Then start it many times, with fixed memory size, using both -Xms and -Xmx.

Example, on Windows:

for /L %i in (1,1,30) do @start cmd /k java -Xms8G -Xmx8G -cp C:\path\to\class Test

On my Windows 7 with 32Gb RAM and no swap file, if I run 8 instances of 4Gb, 3 of them fails with:

Error occurred during initialization of VM
Could not reserve enough space for object heap

That's to be expected, since both OS and other programs were already using memory. I could see the "Commit Charge" maxing out in the Windows Resource Monitor.

So, try this and monitor the OS resources while you do. And of course adjust above command as needed if not using Windows.


the8472 February 2016

The default collector is multi-threaded and its number of threads is scaled based on the number of CPU cores. If you run many java instances at once and with only one thread each you may want to switch to the serial collector, that'll consume fewer threads and virtual memory for thread stacks.

Additionally, the JVM reserves a lot of virtual memory up-front, potentially more than it will actually need during itslifetime. So you should enable swap and allow overcommit to avoid resource exhaustion.

Post Status

Asked in February 2016
Viewed 2,350 times
Voted 5
Answered 2 times

Search




Leave an answer