JVM crashes when attempting to run multiple invocations on a high-memory, many CPU server
I've written some Java-based, single-threaded, genomics software that runs on one sample at a time, and I have hundreds of samples to process. I have access to a machine with 64 CPUs and a terabyte of RAM, and there are no other users on the system. The maximum heap size requested for each invocation is 8 gigabytes. I expect I should be able to invoke 30 instances of my code simultaneously (assuming two threads - my main thread and a GC thread?). I'm only attempting to dispatch 20 at a time (using a makefile and the -j20 argument). In practice, however, only 5 run. The rest fail with the message:
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
Suspicious that this had more to do with the simultaneous invocations than actual resource limitations, I implemented a random several-second delay in my execution loop for each invocation of my program. That gets me up to 10 programs running simultaneously, rather than 5, with the same failure message.
Why does attempting to invoke dozens of instances of the JVM simultaneously fail this way, despite doing so on a system that clearly has the resources available?
Why does my hack implementing the dispatch delay clear up some of the problem?
What's a better way to get all 20 instances running simultaneously?
The default collector is multi-threaded and its number of threads is scaled based on the number of CPU cores. If you run many java instances at once and with only one thread each you may want to switch to the serial collector, that'll consume fewer threads and virtual memory for thread stacks.
Additionally, the JVM reserves a lot of virtual memory up-front, potentially more than it will actually need during itslifetime. So you should enable swap and allow overcommit to avoid resource exhaustion.
Asked in February 2016Viewed 2,350 timesVoted 5Answered 2 times