Creating a big data export

mrkitt

One of our customers is asking to send logs from August to present day. However, they have different sources of logs and some of them such as Windows events and EDR logs have over 150 millions of entries in SIEM. This is extremely huge amount of data to send and I believe ELS server may break from this task. I came to just sending small portions of multiple data exports, but I wonder if there is a best way to send huge amount of data from different sources. Note: I know we can just send a compressed archive of logs, but I suppose this is not what are they expecting to receive.

SzymonC

Hi,

Do you have to deliver the data in some specific format? I mean can it be json? Or does it have to be csv?
Is the data now in archive or is it "online", meaning it is existing in open indices of the cluster?

I'm not sure how comfortable are you with more technical questions, so if you're not sure the answer, let me know, I'll explain a little more. 🙂

mrkitt

SzymonC They have not mentioned a certain format, but I suppose it better be csv. Data is available in open indices, so I can read them in Discover.

SzymonC

Well, in my honest opinion - csv/xlsx documents were never meant to be operating on such large data sets. That's why we have and use databases.
While I don't see issues (other than time needed) with exporting data from ELS, finished file in itself will be most likely useless - due to sheer size. I expect Excel or other programs to simply crash or refuse to operate on it. Quick googling provides some insight: https://csvloader.com/csv-guide/understanding-csv-and-row-limits

So what can we do?
In such case I'd try to get as much info from the Customer what is exactly expected from that report. Maybe they are interested only in specific data? Maybe they don't need to access every raw data and would be interested in statistics/analytics from ELS? If export is literally what Customer wants, then I'd consider two options:

Create a dump out of the index with /usr/share/kibana/elasticdump/elasticdump program. ( https://github.com/elasticsearch-dump/elasticsearch-dump )
Export the data with snapshot method (which isn't that popular these days): https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/ Keep in mind that snapshot still has format unique to the database, so similar database would be needed to open snapshots.

mrkitt

Ok seems like we do not have to worry about a format, so I can create an archive of an index. However, is it ok to create such a huge task, because I am going to create one big archive which consists data from August till present

SzymonC

Great.
Archive can handle long-lasting tasks without any problem. It's hard to estimate how long it will take in your environment, depending on your cluster performance. In general for medium, large to super large environments it's highly recommended to place one dedicated machines for archiving only (if you don't have that already). Difference in cluster performance is significant and such approach doesn't require change in ELS licensing, since archiving machine isn't counted as data node.

That being said - I'm not sure what is the size of those indices in GB/TB. If we are talking about 500 GB, then I'd run that in single job. If it's 1-2TB or more, then maybe I'd split it into couple of tasks (2, 3 or more). How do we do that?
Let's say that we have data from 120 days that we want archive.

First you create a task that will archive data older than 30 days and wait for it to finish.
Second (without removing that archive from archive space) you can create second task, that will archive data older than 60 days. Archive will recognize that the data from 30 days is already archived, so it will focus on data 30 days+ old.
Third you can repeat the process for 90 days, and same for 120 day old data.

In that way you have the compromise of having tasks that will end in reasonable time individually and will produce compressed files that that later Customer can open. But, if you are patient enough you can run everything in single run, it will only take longer time.

Here's example how you can configure single running task:

Submit will produce warning, but you can ignore it:

Here's where you click to run it manually (note that controls will appear when you hover your cursor over them):

If you'll decide to run everything in a single task and it will take long time, then there's also a way to see what is the progress of that task, but it require console access to linux server and checking metadata files. You can write back here and I'll show you how to do that.
In future versions of Energy Logserver that information will be displayed in the archive UI, but it's not available just yet.

Good luck. Let me know if you'll need any further help or if you'll have more questions! 💪