Hi,
Thanks for reaching out about your issue with archiving the 630 GB winlogbeat merged index! I understand you’re having trouble with the merged index, but daily split indexes work fine. I’ll address all your questions based on the Energy Log Server documentation and changelog to help you tackle this.
- Can you provide us information of the location of the logs related to failed archiving?
The documentation doesn’t pinpoint a single spot for archiving error logs, but the changelog gives us some clues:
Elasticsearch Logs: Since archiving relies on Elasticsearch, errors like connection issues or timeouts might show up in /var/log/elasticsearch/ (e.g., elasticsearch.log). Look for terms like "archive" or "connection." Section 16.15.3 mentions fixes for "exception during connection problems to Elasticsearch," suggesting these logs are relevant.
Kibana Logs: The Archive plugin runs within Kibana, so related issues (e.g., GUI or API errors) could be in /var/log/kibana/ (e.g., kibana.log). Changelog section 16.15.2 notes "improved logs verbosity" for archiving since version 7.0.5, making this a good place to check.
Diagnostic Tool: For a full log collection, use the support-tool.sh script (formerly diagnostic-tool.sh, renamed in 7.1.1, section 16.12.2) located at /usr/share/kibana/utils/. Run:
/usr/share/kibana/utils/support-tool.sh
This bundles logs from Elasticsearch, Kibana, and other components, including archiving-related ones.
Recommendation: Start with /var/log/elasticsearch/ and /var/log/kibana/. If you need a broader view, run support-tool.sh.
- Do we need manually in Logstash to configure splitting the indexes by day regardless of their size, and never merge them into single?
It seems your winlogbeat data is going into a single index (e.g., winlogbeat) rather than daily ones (e.g., winlogbeat-2025.04.01), causing the large merged index. Energy Log Server doesn’t automatically merge indexes unless configured to do so via Logstash or an index policy. To split them by day:
Update your Logstash output config to use a date pattern, e.g.:
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "winlogbeat-%{+YYYY.MM.dd}"
}
}
This creates daily indexes, keeping their size manageable and aiding archiving, as you’ve seen with split indexes.
Without this, merging won’t happen unless explicitly set up, so yes, configuring Logstash this way is needed to avoid a single index.
Recommendation: Yes, set Logstash to split indexes daily—it’s the best way to prevent large merged indexes.
- How should I archive the aggregated index?
The Archive plugin (introduced in 7.0.4, section 16.16.1) is designed to handle index archiving, including your 630 GB winlogbeat. Here’s how:
Setup: In Kibana, go to the Archive section (usually under Config or a dedicated tab). Configure the storage directory in kibana.yml with high compression (section 16.15.1), e.g.:
archive.compressionOptions: { level: 9 }
Execution: Select the winlogbeat index and use the “Run now” option (added in 7.1.0, section 16.13.1). Enable document sorting and direct Elasticsearch connection (section 16.15.2) for better performance.
Large Indexes: If it fails due to size, split the index into smaller chunks using the _reindex API, e.g.:
POST _reindex
{
"source": { "index": "winlogbeat" },
"dest": { "index": "winlogbeat-2025-04" },
"query": { "range": { "@timestamp": { "gte": "2025-04-01", "lte": "2025-04-30" } } }
}
Then archive these smaller indexes separately. Changelog section 16.15.2 mentions optimizations for large files, but 630 GB might still need splitting.
Recommendation: Use Archive with compression and direct connection. Split the index with _reindex if it’s too big.
- How should I disable the merging indexes in Energy Log Server without changing the configuration of Logstash?
If Energy Log Server is merging indexes, it’s likely due to an index management policy rather than Logstash. Here’s how to stop it:
Index Management: In Kibana, under Index Management (introduced in 7.1.0, section 16.13.1), check for a policy on winlogbeat* with “merge” or “forcemerge” actions. Edit it to remove those steps or disable the policy entirely.
Rollover: In Config > Settings (section 16.15.1), ensure rollover settings don’t consolidate indexes—set them to create daily ones (e.g., winlogbeat-%{+YYYY.MM.dd}).
Curator: Review Curator config at /usr/share/kibana/curator/ (updated in 7.1.1, section 16.12.3). In curator.yml and actions.yml, remove or comment out forcemerge actions for winlogbeat*.
Recommendation: Disable merging policies in Index Management or Curator. Verify new indexes stay split afterward.
Summary:
Logs: Check /var/log/elasticsearch/, /var/log/kibana/, or use support-tool.sh.
Splitting: Configure Logstash for daily indexes.
Archiving: Use Archive, split if needed.
Merging: Turn off policies in Index Management or Curator.
