Gen Ohta February 2016

How do you get logstash to output to elasticsearch on a private ip?

Unfortunately, Logstash is not attempting to output to Elasticsearch at the correct IP address. Elasticsearch is configured at a private ip address on port 9200. Logstash is attempting to output to Elasticsearch at localhost:9200. This is shown in the log below.

{:timestamp=>"2016-02-08T16:27:58.572000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200/\"]', but Elasticsearch appears to be unreachable or down!", :client_config=>{:hosts=>["http://localhost:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :error_message=>"Connection refused", :class=>"Manticore::SocketException", :level=>:error}

My configuration files are below: /etc/elasticsearch/elasticsearch.yml

network.host: PRIVATE_IP_ADDRESS

/opt/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["PRIVATE_IP_ADDRESS:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I have the same file at /etc/logstash/conf.d/logstash.conf because I didn't know where to put the logstash configuration file.

When I run curl PRIVATE_IP_ADDRESS:9200, I get the following output

{
  "name" : "Gabriel Summers",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.1.1",
    "build_hash" : "40e2c53a6b6c2972b3d13846e450e66f4375bd71",
    "build_timestamp" : "2015-12-15T13:05:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.3.1"
  },
          

Answers


kkwoker February 2016

I would double check that logstash is actually using that configuration file.


Mukrram Rahman February 2016

Just use follwing snippet to you conf file and it will work.

output {
  elasticsearch {
     hosts => "PRIVATE_IP_ADDRESS:9200"
  }
}


Gen Ohta February 2016

It turns out that when I was restarting logstash all of the processes weren't being terminated. My config files weren't the problem.

Here are the commands I ran to solve this: ps -ef | grep logstash sudo kill EACH_PID sudo kill -9 PID_THAT_WASNT_KILLED_BEFORE sudo /etc/init.d/logstash start


devlearn February 2016

If you use the RPM installation then your configuration should be located under /etc/logstash/conf.d/.

So try to move the file you mentioned ( opt/logstash/conf.d/logstash.conf ) into the /etc/logstash/conf.d directory then restart your logstash service and check it out.

Post Status

Asked in February 2016
Viewed 1,579 times
Voted 5
Answered 4 times

Search




Leave an answer