Logstash config to Postfix mail server

Diaz

Hello! We've integrated the Logstash configuration file for the Postfix mail server from the ./install/integration/syslog-mail/conf.d/syslogmail/ directory. We're receiving logs from the mail server, but the data comes in an unreadable format, and we're getting a message in the service [WARN][logstash.codecs.line][syslog-mail][d6b0a6ff93849f8dbbefac9c34ce78dc1ec924ebda45bc8e262b8b8c63ecd598] Received an event that has a different character encoding than you configured. {:text=>">\u0003\u0019\xDFT\u0006\b\x9F\x96\u0001Lq-\x8F\xCB(\x8F\x93ȭ\xE9\xE435\xD7X\AE.\x8A\xF3\x98\xE0\xD2=\x916G\u0019Z\xB5\xC7\u0004 b\xB8\x9Er&?U\xC8X\u0001\x90\u060AM\xEF\xB7\CF9|C'\e\x8E\x90sw\BA\e\xB2\u001Eq\xE3\x9C1qb\xE3C3\x86\xBBJM.\xBC8\xEFhG\u000F\xF6\xB6\AF&\x89J\u0000_{újc\xED\xC00\xF6\xA3\xC1\x86\xF69\DE5M\u001D\xDF\F9Я\FE\E8\F3\u0005\xE6\xDF}'>c\xC3\xE5\t\u001E=\u0004Ys\E3\u00B4K\u0002\xDAO\xA0l\BF-\x9C\xC6S3\x9E\xB1W\u0000]\xDF\E4\B74\ED\u0089ftnh\xF7\xA5\x95\"j\u0006Q\x87\BF<߿2\x93VE\x96p՝ZHM\u007F\"\xE5`\x9B\x8DA". This is how they appear in Elasticsearch.

SzymonC

Hi,

How does the raw log files from postfix look like? Is there other alphabet than Latin used there?

Diaz

SzymonC We found what the issue was: the parser was written to accept syslog, but we were sending data via the Filebeat agent. We have corrected this, and now we are sending messages in syslog format, and the data is being read. However, the logs are not being parsed correctly, and in the tags, we see _grokparsefailure. It looks like this.
Additionally, Logstash is reporting the following error

SzymonC

Interesting. So let's tackle one problem at the time.

_grokparsefailure occurs often. Though it shouldn't, if it's present it doesn't necessary mean anything serious. It's really tough to create error-free universal parser. 🙂
From what i see the parsing isn't terrible. You can ignore this tag, or remove it in parsing. Question is what else would you like to extract from syslog_message field?
If you want to correct parsers to suit your needs better, you can do it in /etc/logstash/conf.d/syslog-mail/.
The console error is saying that parser is trying to save parsed document in the index, which name has ,(comma) sign in it. Like log is saying, such signs like , " > | / ? etc. are forbidden in index name, thus we have an error.
Where is it coming from? My guess is that you can have multiple values in one field.
If you look at the output file of you syslog-mail parser pipeline, you will see something like this: index => "%{[@metadata][_index]}-%{+YYYY.MM.dd}". The {[@metadata][_index]} is a reference to a field's value from currently parsed document. If you prefer you can move it to something more strict, without such variables.
For example:
index => "syslog-net-postfix-%{+YYYY.MM.dd}"

Let me know if it helped. 🙂