Bilguun February 2016

How to create index on tomcat log file using LogStash

I'd like to have some daily analysis from tomcat log file such as how many errors and exceptions raised and categories of them etc. So that I choose ELK to do that and am new to the log indexing.

Here is my conf file:

input {
    file {
        path => "\localhost.2016-01-09.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout {codec => "rubydebug"}
    elasticsearch {
        hosts => ["localhost:9200"]

    }
}

And here is some lines of the log file:

09-Jan-2016 18:30:38.722 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:30:38.796 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\0-contact-statecollab-ws-15.12-SNAPSHOT-unknown-20151230-1152\WEB-INF\log4j.properties]
09-Jan-2016 18:30:38.901 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
09-Jan-2016 18:30:54.271 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:30:54.316 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\1-ohsms-ws-15.10.16-SNAPSHOT-unknown-20151119-1832\WEB-INF\log4j.properties]
09-Jan-2016 18:30:54.361 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
09-Jan-2016 18:31:14.627 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log ContextListener: contextInitialized()
09-Jan-2016 18:31:14.628 INFO [localhost-startStop-1        

Answers


Bilguun February 2016

I found the problem. I had to provide sincedb_path property(sincedb_path => "\null") in a input -> file property. Because elastic search thought that this index was already created and no need to create it again. Otherwise, ES waits until new lines added to the log file.

Post Status

Asked in February 2016
Viewed 2,963 times
Voted 13
Answered 1 times

Search




Leave an answer