Problem on collector logstash

amir

Hello!

When restarting the logstash collector, the logs start working and stop at some point (usually after 2-3 minutes). When I restart it, it starts up again and stops again. And so on again.

[ERROR][logstash.outputs.logserver][beats][e6c0219f3c0fdfec08a47d19abeebece0209a9501376f78bfc09d76722489d74] Attempted to send a bulk request but OpenSearch appears to be unreachable or down {:message=>"OpenSearch Unreachable: [http://logstash:xxxxxx@10.1.102.27:9200/][Manticore::SocketException] Connection reset", :exception=>LogStash::Outputs::Logserver::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>64}

SzymonC

Hi,

Those errors usually happens when network refuses to send packets from parser to database.
In some cases parser might try to sent bulk packets to data node, if it doesn't have enough performance or the bulk is to big. Such case might be cause when we have data that is unusually large or database doesn't have enough performance at the moment to process it.
What are the details of your environment?

amir

There are no special functions, I have 3 nodes, and there are separate logtash data collectors on the VM. The problem is that there are problems with the rest of the log indexes, the problem is all in beats

SzymonC

I see. To verify if the problem is with that pipeline alone we can create simple and empty pipeline for beats in order to verify if this would occur without parsing. Of course that would mean that for some period we wouldn't have parsed data (only the original format from the source). Would that be a viable option for you or that data stream is to critical to interfere with? If we cannot temporary disable filtering/parsing, then there are other options, but above i think might be the simplest.
Also - is data stream for beats is only in latin or other signs, like cyrylic alphabet?

amir

I've already removed filtering/parsing, it didn't help. Beats has a Latin data stream. I also forgot to tell you that it started after I connected auditbeat (the same as filebeat), is it possible because of this?

[ERROR][logstash.outputs.logserver][beats][e6c0219f3c0fdfec08a47d19abeebece0209a9501376f78bfc09d76722489d74] Attempted to send a bulk request but there are no living connections in the pool (perhaps OpenSearch is unreachable or down?) {:message=>"No Availble connections", :exception=>LogStash:😮utputs::Logserver::HttpClient:😛ool::NoConnectionAvailableError, :will_retry_in_seconds=>2}

[ERROR][logstash.outputs.logserver][beats][e6c0219f3c0fdfec08a47d19abeebece0209a9501376f78bfc09d76722489d74] Attempted to send a bulk request but OpenSearch appears to be unreachable or down {:message=>"OpenSearch Unreachable: [http://logstash:xxxxxx@10.1
.102.27:9200/][Manticore::SocketException] Connection reset", :exception=>LogStash:😮utputs::Logserver::HttpClient:😛ool::HostUnreachableError, :will_retry_in_seconds=>64}

SzymonC

It is possible, if parser tries to ingest it into other type of beat parsing. Did you add auditbeat in the auditbeat config tags?

The best way to verify, if possible, is to temporary deactivate auditbeat and see if errors appear.

amir

How will adding a tag help me?

yes, I temporarily disabled auditbeat and the logs went. Does the Energy log server even support the auditbeat plugin? And can port 5044 block the sending of logs due to too much data?

SzymonC

In Energy Logserver tags are often used for parser to make sure data goes through right parsing rules. I'm not sure if there are predefined parsing rules, but it's always good idea to tag data from beats by agent name, so data from winlogbeat agent with winlogbeat tag, filebeat agent with filebeat tag, metricbeat with metricbeat tag, and so on.

Yes, Energy Logserver supports data from auditbeat, even if we would assume there's no predefined parser for it, that data from all beats agents should be sent on port 5044.

Important thing to keep in mind while installing beats agents is to use them in open-source (OSS) version and with proper version. For your version, if I'm not mistaken 7.17 is the right one. You can check it in the Agents module in the UI. If you click on the top right button on the Agents Download, you'll see for sure.

So let me get this right - after deactivating data flow from Auditbeat the errors stopped appearing?

amir

Yes, after disabling auditbeat. The logstache started working normally.

I also wanted to replace the fact that for some reason it eats up too much RAM. And it gives the error that I directed above.

SzymonC

OK, so actually we have couple of targets here to tackle 🙂

Right, let's start with the auditbeat.

I'd verify if adding tag to the agent config would help solving this. Also - check if it's correct version with OSS deployment.
In case of errors appearing again, in order to debug this, I'd create simple dedicated pipeline in /etc/logstash/pipelines.yml and send data through it, in order to verify if the problem would still occur.
2.1 Dedicated pipeline in very easy to do, because we actually need just input, empty filter, and simple output. So everything should look something like this (note that we need different, unused port):
```
input {
  beats {
    port => 5099
  }
}
filter {}
output {
  logserver {
    hosts => [ "127.0.0.1:9200" ]
    index => "auditbeat-test"
    user => "${LOGSTASH_USER}"
    password => "${LOGSTASH_PASS}"
  }
}
```

Now, going to the RAM usage by parser, you can control that in /etc/logstash/jvm.options file. There's a section as below:

 ## JVM configuration

 # Xms represents the initial size of total heap space
 # Xmx represents the maximum size of total heap space
  
 -Xms1g
 -Xmx1g

Xms is minimal amount of RAM that jvm can take from system to run the parser, while Xmx is maximum. In above example it's 1GB memory. We want to have both those values the same.
Same file also exists in database configuration: /etc/elasticsearch/jvm.options.

Here are example of simple calculation:

Take the whole OS memory and subtract from it RAM that is needed for other services, OS and parser. So if you have 32 GB RAM in your system, subtract from it (for this example) 6 GB of RAM.
32GB - 2G (parser) - 4 GB (OS) - 0 GB (other services) = 26 GB left
Divide by 2 what's left from above calculation.
26 GB / 2 = 13 GB
13 GB is the maximum number of RAM that we can assign to ELS database jvm for most stable performance.

Hope this helps. 🙂

amir

It didn't help(

SzymonC

But those errors are regarding beats pipeline. We can see it here:
[ERROR][logstash.outputs.logserver][beats][e6c0219f3c0fdfec08a47d19abeebece0209a9501376f78bfc09d76722489d74]

So if you've changed port on your auditbeat, the problem is not related with this agent or by accident you've sent the data to the same port, thus same pipeline.

We can further work this out here, but remember that there's always an option to reach out to your local distributor for assistance on this from our engineers during online session. It's up to you. 🙂

amir

I changed it as you asked for a new auditbeat config for port 5099, it didn't help(

Logging goes on and then stops after a while

[ERROR][logstash.outputs.logserver][auditbeat][e94ffb8efc216a7150ce2f221ba13a21cb44711eda309956c406d4af6de79db9] Attempted to send a bulk request but there are no living connections in the pool (perhaps OpenSearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash:😮utputs::Logserver::HttpClient:😛ool::NoConnectionAvailableError, :will_retry_in_seconds=>64}

SzymonC

Set the output to a flat file, instead of logserver database. Let's see what it's actually sending.
Also, what is your version of ELS? You can check it in the top right corner after you log in.

amir

Version 7.4.2,

I did not understand you how to check this?

SzymonC

The auditbeat pipeline now is sending data to ELS database using logserver plugin in the output section. To verify what and how many data we can do a simple switch, so parser will save this data to a flat file, instead of sending it to the database.
For that we can change output plugin to file. It should look something like that:

output {
 file {
   path => "/var/lib/logstash/test-output-file.txt"
 }
}

And for the version, update maybe will help as well. Current version is 7.5.0.

amir

Yes, there are logs in this format:
{
"system": {
"audit": {
"package": {
"summary": "Tag Image File Format (TIFF) library",
"version": "4.5.0-6+deb12u1",
"arch": "amd64",
"size": 711,
"name": "libtiff6",
"entity_id": "REDACTED",
"url": ""
}
}
},
"tags": ["linux-server", "beats_input_codec_plain_applied"],
"agent": {
"type": "auditbeat",
"version": "7.17.11",
"name": "REDACTED",
"id": "REDACTED",
"ephemeral_id": "REDACTED",
"hostname": "REDACTED"
},
"service": {
"type": "system"
},
"ecs": {
"version": "1.12.0"
},
"package": {
"architecture": "",
"type": "dpkg",
"version": "4.5.0-6+deb12u1",
"size": 711,
"description": "Tag Image File Format (TIFF) library",
"name": "libtiff6",
"reference": ""
},
"@timestamp": "2024-07-18T09:19:46.559Z",
"@version": "1",
"message": "Package libtiff6 (4.5.0-6+deb12u1) is already installed",
"host": {
"architecture": "x86_64",
"os": {
"type": "linux",
"family": "debian",
"version": "12 (bookworm)",
"platform": "debian",
"kernel": "6.1.0-18-amd64",
"name": "Debian GNU/Linux",
"codename": "bookworm"
},
"containerized": false,
"mac": ["REDACTED"],
"name": "REDACTED",
"id": "REDACTED",
"ip": ["REDACTED"],
"hostname": "REDACTED"
},
"event": {
"type": ["info"],
"dataset": "package",
"category": ["package"],
"module": "system",
"kind": "state",
"id": "REDACTED",
"action": "existing_package"
}
}

amir

Maybe that's the problem?

SzymonC

Hi amir,

Structure looks ok. Is this pipeline throwing any errors as it is writing to a file or is it working stable under this configuration?

amir

There is no error, it works stably with this configuration.