Unknown fields

anonymous

I want to create a visualization with certian critieria, however some fields in my index are unknown. I have already tried to refresh the field list in management => index patterns, but it did not help. I have also read about that mapping needs to be enabled to make elasticsearch parse those fields, but I want to make sure that everything I do will not cause data loss. Here is the mapping settings for my index:

{
"alert-2023.06.04-000003" : {
"mappings" : {
"dynamic_templates" : [
{
"message_fields" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "keyword"
}
}
}
],
"date_detection" : true,
"numeric_detection" : false,
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "date_optional_time"
},
"@version" : {
"type" : "keyword"
},
"aggregate_id" : {
"type" : "keyword"
},
"alert_id" : {
"type" : "keyword"
},
"alert_info" : {
"properties" : {
"command" : {
"type" : "keyword"
},
"hive_host" : {
"type" : "keyword"
},
"risk_value" : {
"type" : "keyword"
},
"type" : {
"type" : "keyword"
}
}
},
"alert_sent" : {
"type" : "boolean"
},
"alert_time" : {
"type" : "date",
"format" : "date_optional_time"
},
"data" : {
"type" : "object",
"enabled" : false
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"kibana_discover_url" : {
"type" : "keyword"
},
"match_body" : {
"type" : "object",
"enabled" : false
},
"match_time" : {
"type" : "date",
"format" : "date_optional_time"
},
"rule_name" : {
"type" : "keyword"
},
"until" : {
"type" : "date",
"format" : "date_optional_time"
}
}
}
},

I need to enable the mapping of match_body field with an object type, as in this object I have a match_body.agent.lables.tenant which is necessary for making visualizations.

SzymonC

Hi,

It's great that you've done your research, kudos to you!
In general match_body enabled parameter is set to false, since it would map every single field for every single field in monitored indexes. You can imagine it would go to thousands quite easily. On the other hand visualizing alerted data is very helpful, so this is what I would do.

Create new index template with higher order, very easy (example below), in which I would specify the list of fields that i want alert index to put into the mapping. Now this will work only for newly created index, so you either wait for new alert-... index or you'd have to play a little with index rollover.

Create new template. I recommend using Cluster view for that (top right corner)

Paste below:

  "order": 90,
  "index_patterns": [
    "alert-*",
    "alert_status",
    "alert_error",
    "alert_silence"
  ],
  "settings": {},
  "mappings": {
    "properties": {
      "match_body.agent.lables.tenant": {
        "type": "object",
        "enabled": "true"
      }
    }
  },
  "aliases": {}
}

Of course, if you have more fields that you know now that will be needed for visualizations - just add them to the list here in the properties section.

Find out the latest alert-.... index. You can do that in the Cluster view.
Rollover alert alias with next number of index. You can use Dev Tools UI module for that:

Ad. 1. I'm pointing here new write index, so it's name must be different. Ideally just 1 number higher.
Ad. 2 Here are the conditions that must take place to create new index for rollover. Note that database is not monitoring this constantly, those conditions must be exceeded at the moment of executing this command. That's why my condition is max_docs is 140, while my old write index has already 146 of them.

And in that way you can create new index that will speed this process up. You can always check the mapping of new index to make sure for 100% if that worked out.

Let me know if it helped. 😉

SzymonC

Also, if for whatever reason you'd delete write index and will stuck, this is a command that assign writing index again, so alerts will continue to be saved in the database:

POST /_aliases?pretty
{
  "actions" : [
    { "add" : { "index" : "alert-2023.09.28-000212", "alias" : "alert", "is_write_index": true } }
  ]
}

Please note that index alert-2023.09.28-000212 must already exists and it cannot be already active write_index.

anonymous

Its been a while but yesterday I created a new index template as you showed. However , nothing changed and fields that I have set stayed unmapped. There are two index templates for alert indices, the default one and custom one that I created:
Default template:

{
  "order": 10,
  "index_patterns": [
    "alert-*",
    "alert_status",
    "alert_error",
    "alert_silence"
  ],
  "settings": {
    "index": {
      "refresh_interval": "1s",
      "number_of_shards": "1",
      "auto_expand_replicas": "1-2",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "aggregate_id": {
        "type": "keyword"
      },
      "alert_time": {
        "format": "dateOptionalTime",
        "type": "date"
      },
      "@timestamp": {
        "format": "dateOptionalTime",
        "type": "date"
      },
      "data": {
        "type": "object",
        "enabled": "false"
      },
      "rule_name": {
        "type": "keyword"
      },
      "alert_id": {
        "type": "keyword"
      },
      "match_body": {
        "type": "object",
        "enabled": "false"
      },
      "match_time": {
        "format": "dateOptionalTime",
        "type": "date"
      },
      "until": {
        "format": "dateOptionalTime",
        "type": "date"
      }
    }
  },
  "aliases": {}
}

The custom one:

{
  "order": 90,
  "index_patterns": [
    "alert"
  ],
  "settings": {},
  "mappings": {
    "properties": {
      "match_body._index": {
        "type": "object",
        "enabled": "true"
      },
      "match_body.agent.lables.tenant": {
        "type": "object",
        "enabled": "true"
      }
    }
  },
  "aliases": {}
}

Maybe changes did not apply because of order priority? Additional question is, should I really set those fields (_index, agent.lables.tenant) as object types, not strings? I tried to set them as type "string", but the service throwed an error.

SzymonC

Well, let's see if templates did work. If so, new alert indices should show them. In the Cluster view, find latest alert index and on the arrow next to it click show mappings.

If template did work, you should see properties section and there you should find your new fields.

is the field name
is the field type

If that works, then operation was successful. That means that most likely all that is left is to refresh alert index pattern, or create new.

First let's make a backup of existing index pattern:
Go to the UI -> Stack Management or Management (depends on the version you are using).

Select Saved Objects
Choose index-pattern type
Add alert to search bar
Select pattern you'll be refreshing (you can select multiple if you're not sure which one is correct)
Click on Export button
Again, click on Export.
This will download ndjson file, which you can use to import and reverse any changes to that object.
Why we need to backup? Well, we don't need to, we want to. 🙂
Refreshing an index pattern is a simple operation which refreshes list of fields inside that pattern (don't confuse it with index. Index and index pattern are two different things). Sometimes, when some old index is deleted, field that has been change or is no longer needed is also removed after such refresh and this might cause some logic errors in the visualizations (like trying to visualize values of the field that no longer exists).

Anyway, now that we have backup of index pattern, we can refresh it and check if this helps. Refreshing is quite simple actually. Just go to the Index Patterns category, find pattern you want to refresh and click refresh icon:

After that refresh, refresh the page and see if you can find your field, or if you can search on in in Discover module, or if you can visualize it.

Hope that helps!

anonymous

On mapping settings I have not seen the necessary fields, after that I enabled mapping on the entire match_body and in fact, this worked. However, as you said, this is not reliable as it would make thousand of fields to index separately. Seems like it is not possible to enable mapping on internal fields of match_body, unless you enable mapping on match_body itself, or there should be another extra step to implement this.. Correct me if I am wrong.

SzymonC

anonymous I think it should work. 🤔I'll run some tests again and will get back to you.

SzymonC

OK, so original "enabled": "false" without any parameters is in fact blocking all further operations on subfields of that field-object (in our case it's match_body).
But there's easy solution. Basically whole procedure stays the same, but:

Edit default-alert-indices. Change parameter enabled on true and add parameter "dynamic": "false". End result should be like this:
Add new template, similar to the one with "order": 90, but add/change the type of fields you'll want to visualize from object to keyword. Like so:
Continue with _rollover command to make sure new index will be created. Check it's mapping after creation. Remember to refresh index-pattern in UI (also for visibility i recommend delete older alert indices if you're on test environment). It works on my tests and I was able to visualize this field.

Let me know here if you'll have any difficulties with setting this up, I'll be glad to help.

anonymous

Big thanks, it helped indeed