Configuring Kibana for an elasticsearch cluster

Diaz

We've set up separate master nodes, totaling three in number. We removed the master role from one of the data nodes, where Kibana is also deployed. So now, our architecture consists of three master nodes and two data nodes, one of which includes Kibana. In the kibana.yml, we've added IP addresses with port 9200 (previously it was 8000). However, Kibana hasn't started and is unresponsive. Could you suggest possible solutions to this issue?

SzymonC

Well, yeah. Port 8000 has different function than port 9200. 😃
If you are talking about option opensearch.hosts: ["http://127.0.0.1:8000"] (or elasticsearch.hosts - depends on the ELS version) in kibana.yml file, it should stay as 8000. Try changing it back and restarting kibana service.

If anything you can check logs in /var/log/elasticsearch/kibana.log or journalctl -fu kibana.

Diaz

We changed the port, but the situation hasn't changed. Additionally, we decided to move Kibana to a separate host, and now it's on a separate node. The command 'journalctl -fu kibana' gives us these logs.

Also, we're unsure if specifying multiple master hosts in the 'opensearch.hosts' section like this is correct: ['http://192.168.1.199:8000, http://192.168.1.151:8000, http://192.168.1.152:8000'] in the kibana.yml file, in order to ensure Kibana accessibility in case the master host fails. How correct is this?

SzymonC

I've run some tests and this is how it works (i know it's a little strange):
opensearch.hosts: ["http://127.0.0.1:8000"] this option should always point to localhost. This is because port 8000 is used for security/authorization functions.

Localhost instance then is communicating with databases using: elastfilter.url: "http://<database_ip>:9200" and here database can be remote. Now, this parameter DOES NOT accept multiple values.
For best performance and user experience i recommend to install small instance of database on the same host as UI.
In the Database config set role.data: false. This will make that this database to act as a client (thus will not require additional license). At the same time make this new client node part of the whole cluster and in result you'll be reasonably protected from connection issues.
As a good advice I can share that we did had deployments where in such cases this "client" database had set role.master: true. It was working good, so this might be an idea to consider.