PeterKA February 2016

Java Running at close to 100% CPU

Too much logging activity over the weekend led to the following error being thrown by ColdFusion:

Message:    No space left on device
StackTrace: java.io.IOException: No space left on device at
            java.io.FileOutputStream.writeBytes(Native Method) at 
            java.io.FileOutputStream.write(FileOutputStream.java:269) at 
            coldfusion.compiler.NeoTranslator. ......

By this morning pages on the ColdFusion web site were not loading at all. The disc (12Gig) was over 99% used up. We moved several files to the second disc and now it's at about 80%, much lower than where it's always been. We're going to direct logging activity to the second disc (100Gig) to prevent a repeat. Having created space on the disc we restarted apache and coldfusion but still pages are not loading.

When we run top -H it appears that java is running at close to 100% CPU. Does anyone have a clue what's going on or what info I need to provide so someone can figure it out?

The set up is AWS, ubuntu 13.04, coldfusion 10, mysql (rds).

UPDATE

I have made some really strange but hopefully helpful observations. I am still trying to locate a tool to help with getting thread dumps. Whenever I restart ColdFusion, most pages load fine and CPU usage seems normal, mostly 0.7 - 1.5% but once in a while spikes of up to 10% can be seen. But there's one particular page, which when I try to load, causes CPU uses to rise to 97% always. The pages that load fine have a simple query reading data from one table. This problematic page has an inner join, and reads data from two tables. I don't know how helpful this is but I think it is too consistent to be insignificant.

UPDATE 2

And since this issue started the following errors appeared for the first time and there're hundreds of lines of it:

[2243:140630871263104] [e        

Answers


Aaron February 2016

You would probably want to provide thread dumps, a few of them taken at a few seconds interval, and gc logs.

Thread dumps can be produced with jstack (a tool provided in your jdk's bin directory), and garbage collector logs must be activated beforehand.

Post Status

Asked in February 2016
Viewed 3,237 times
Voted 12
Answered 1 times

Search




Leave an answer