TCP(/logserver?) output blocked - other outputs also blocked

MrAlpaca

Hi, are these links still relevant to the Network Probes?

https://github.com/elastic/logstash/issues/8524
https://discuss.elastic.co/t/logstash-tcp-output-blocks-entire-pipeline/85818

If I undestand it correctly - if one of our pipelines has many outputs its enough that one of them is tcp (logserver?) output and other are also blocked?

I see similar behaviour in one of our pipelines where we have only logserver outputs to one ELS cluster and udp outputs to secondary siem and when there is problem with logserver output (index with that alias is recovering or blocked by other reason), output via udp to other server also stops, is this the same situation?

SzymonC

Hi,

Generally - yes. That's why it's always good idea to spread your data on multiple pipelines. That's also the reason why we often use two logstash instances wehen ELS architecture is more distributed:

Layer 1 as data receiver from sources
Layer 2 as main parser and receiver from L1 logstash.

Between those two layers we place Apache Kafka, which allows us to avoid this situation. Kafka cluster can keep data-in-transfer between two servers and protects it if anything fails.