Archive module doesn't archive

mrkitt

There is an index which has a significant number of logs during 2023. The plan was to archive those indexes into SAN and then delete from SIEM.

As you can see in the picture above, an index pattern in discover shows logs presense

After archiving, it just archived only one of the many indexes (archived properly for date 2023.05.03), but for others, it just created an empty .json.zstd file which is 4.0 kilobyte in size
[root@siem_els 05]# du -sh winlogbeat-pvl-2023.05.03_2023-05-03.json.zstd 6.1G winlogbeat-pvl-2023.05.03_2023-05-03.json.zstd [root@siem_els 05]# du -sh winlogbeat-pvl-2023.05.02_2023-05-02.json.zstd 4.0K winlogbeat-pvl-2023.05.02_2023-05-02.json.zstd [root@siem_els 05]# du -sh winlogbeat-pvl-2023.05.01_2023-05-01.json.zstd 4.0K winlogbeat-pvl-2023.05.01_2023-05-01.json.zstd
If I restart the task for archiving, it just immediately turns to a complete status

SzymonC

Hm...
While archive task is going, it creates a temporary file that tracks the progress of archiving. Go to there directory: /usr/share/kibana/data/archive/tasks and check the .json file to verify what's current status and if there are any errors.

Also it's worth to keep in mind that for bigger archive tasks it's highly recommended to have a dedicated archive node (if you don't have that already) responsible for archiving only. Basically it's specialized GUI node. You can configure roles of the UI by enabling/disabling specific functions in the /etc/kibana/kibana.yml. In this case its: archive.enabled: false/true.
Dedicated archive node will not impact licensing of ELS in common licensing model.

mrkitt

I was able to recreate the problem on a test server and solve it, but it is too risky for doing on production and there are many archives that we may lose if something goes wrong, so I wonder if there is a better solution. Let me explain why archives couldn't be created for some indexes: If you create an archive for index ABC successfully, then delete it from a console, Kibana shows that file can't be found in directory, like in the screenshot

If you try to create an archive again, ELS will create an empty archive with no data or doesn't create it at all

The solution is to delete indices ".archive" and ".archive_task" which contains metadata about archive files. After restarting kibana, archives were created without problems. However, deleting .archive affects all archives in ELS, including normal ones, afterwhich kibana says that there is no metadata.

SzymonC

I've reach out to our Dev team and here's their help on this subject (thanks guys!):
To work around this confirmed bug, there is no need to delete either the .archive_task index or whole .archive index. Follow the following steps.

Search for the metadata of the archive that has no corresponding .zstd file, using following query. Replace ARCHIVE_NAME with the name of the file displayed in the Search/Restore tables:

POST .archive/_search?q=filename:ARCHIVE_NAME

Get the _id of the retrieved metadata document and remove the document itself by executing query below. Replace ID with the found _id.

DELETE .archive/_doc/ID
Restart of the kibana service is not required.
If there is any suspiciously small .zstd file, remove it as well from the storage.

Retry archivisation process. Now the documents will be rediscovered and newly created archive should consist of correct file and metadata.

mrkitt

SzymonC any updates on this?

mrkitt

Thanks it helped indeed, but here is another problem: when I restore the archive, kibana stops responding in browser even though service is active.

Because that server was a part of production, I also tried to move the archive to another ELS cluster and restore it there (it warns that there is no metadata related to this particular file), and after that, I got an error

SzymonC

Hi, sorry I was on vacation. 🙂
So I've discussed it internally with devs and we have few theories.

First let's start with UI service restarting - is this happening every time or was it just one time event?
Second - did you had chance if the archives can be opened/decompressed without ELS archive function? I mean we are using Zstandard algorithm, so you could bypass UI to open it. https://en.wikipedia.org/wiki/Zstd
Third - If the archives has been damaged during archiving process, it'd be logical to encounter errors during decompression. You could try restoring archive with CURL command from the console. It would look somewhat like this:
curl -XPOST -k https://localhost:5601/api/archive/restore_task/run?debug=true -H 'osd-xsrf: true' -d '{"files": ["apm-7.2.1-span-2023.11.20_2023-11-20.json.zstd", "apm-7.2.1-span-2023.11.21_2023-11-21.json.zstd"] , "verifyChecksum": false, "encryptionPassword": null, "initiatedUser": "logserver", "destinationIndex": "apm-span-restore-20-22" }'
In result you'll find task in /usr/share/kibana/data/archive/tasks and .debug.log

Let me know if this helps.

mrkitt

UI is not restarting, it just stops responding to any clicks, that is why I had to kill the restoring process and restart kibana

Archives themselves are ok, I was able to decompress them with zstd. The problem is that when I copy an archive to our test ELS, on restore page of UI it shows that there is no metadata regarding this file, and eventually it fails to decompress

SzymonC

OK, what are the sizes of those archives files?

mrkitt

varies from 1.5GB to 4GB, one decompressed state it could be around 100GB or more

SzymonC

Interesting. What are the versions of both of the environments? Also could you check/send kibana logs? If they are not present /var/log/elasticsearch/, you can extract it from journalctl -u kibana.