Sometimes Logstash shows these kind of errors and stops sending logs:
Mar 26 09:06:30 logstash-host logstash[33219]: [2024-03-26T09:06:30,166][ERROR][logstash.outputs.logserver][beats][2052526f092bed8ac5bb74c8a9e32c5081d8f44efab884ea699c900b17c6e387] Attempted to send a bulk request but there are no living connections in the pool (perhaps OpenSearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::Logserver::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
Mar 26 09:07:06 logstash-host logstash[33219]: [2024-03-26T09:07:06,630][ERROR][logstash.outputs.logserver][beats][2052526f092bed8ac5bb74c8a9e32c5081d8f44efab884ea699c900b17c6e387] Attempted to send a bulk request but OpenSearch appears to be unreachable or down {:message=>"OpenSearch Unreachable: [http://logstash:xxxxxx@10.1.100.53:9200/][Manticore::SocketException] Connection reset", :exception=>LogStash::Outputs::Logserver::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>64}
A temporary solution was to restart logstash, but after some time this error happens again. An interesting part is that data is not sent only on beats pipeline, others are fine and work without any problems
Logstash winlogbeat pipeline:
output {
if "winlogbeat" in [tags] {
logserver {
hosts => ["10.1.100.53:9200"]
index => "winlogbeat-%{+YYYY.MM.dd}"
user => "logstash"
password => "logstash"
ilm_enabled => false
}
}
}
Filebeat pipeline:
output {
if "filebeat" in [tags] {
logserver {
hosts => ["10.1.100.53:9200"]
index => "filebeat-%{+YYYY.MM.dd}"
user => "logstash"
password => "logstash"
ilm_enabled => false
}
}
}
