Update elastic/security artifacts

[achuguy]

Update the artifacts, composable templates, component templates, pipelines, ilm, used in the elastic/security track to Elasticsearch 8.15.1. Also deleting unused artifacts from the repo.

[salvatore-campagna]

@achuguy Under composable-logsdb there are copies of some files that are specific for the logsdb benchmark. Can we make sure the changes are applied to those files too? Thank you.

[salvatore-campagna]

Question: does that logic generating the dataset needs adjustment?

[achuguy]

Question: does that logic generating the dataset needs adjustment?

For generating a dataset I'll take endpoint data from a cluster with real endpoints. There is room to adjust fields in the dataset depending on cardinality needed.

[achuguy]

@achuguy Under composable-logsdb there are copies of some files that are specific for the logsdb benchmark. Can we make sure the changes are applied to those files too? Thank you.

@salvatore-campagna When I'm trying to run the security-track with track-param "index_mode": "logsdb" I get an error trying to create the index template:

Create composable index template

PUT _index_template/logs-endpoint.events-process-test
{
    "index_patterns": [
      "logs-endpoint.events.process-*"
    ],
    "template": {
      "settings": {},
      "mappings": {
        "_meta": {
          "package": {
            "name": "endpoint"
          },
          "managed_by": "fleet",
          "managed": true
        }
      }
    },
    "composed_of": [
      "logs@mappings",
      "logs@settings",
      "logs-endpoint.events.process@package",
      "ecs@mappings",
      ".fleet_globals-1",
      ".fleet_agent_id_verification-1",
      "track-shared-logsdb-mode"
    ],
    "priority": 200,
    "_meta": {
      "package": {
        "name": "endpoint"
      },
      "managed_by": "fleet",
      "managed": true
    },
    "data_stream": {
      "hidden": false,
      "allow_custom_routing": false,
      "failure_store": false
    }
  }

Error

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[37:7] [data_stream_template] unknown field [failure_store]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[37:24] [index_template] failed to parse field [data_stream]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[37:7] [data_stream_template] unknown field [failure_store]"
    }
  },
  "status": 400
}

Is the failure_store parameter correct under data_stream?

[salvatore-campagna]

@achuguy Under composable-logsdb there are copies of some files that are specific for the logsdb benchmark. Can we make sure the changes are applied to those files too? Thank you.

@salvatore-campagna When I'm trying to run the security-track with track-param "index_mode": "logsdb" I get an error trying to create the index template:

Create composable index template

PUT _index_template/logs-endpoint.events-process-test
{
    "index_patterns": [
      "logs-endpoint.events.process-*"
    ],
    "template": {
      "settings": {},
      "mappings": {
        "_meta": {
          "package": {
            "name": "endpoint"
          },
          "managed_by": "fleet",
          "managed": true
        }
      }
    },
    "composed_of": [
      "logs@mappings",
      "logs@settings",
      "logs-endpoint.events.process@package",
      "ecs@mappings",
      ".fleet_globals-1",
      ".fleet_agent_id_verification-1",
      "track-shared-logsdb-mode"
    ],
    "priority": 200,
    "_meta": {
      "package": {
        "name": "endpoint"
      },
      "managed_by": "fleet",
      "managed": true
    },
    "data_stream": {
      "hidden": false,
      "allow_custom_routing": false,
      "failure_store": false
    }
  }

Error

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[37:7] [data_stream_template] unknown field [failure_store]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[37:24] [index_template] failed to parse field [data_stream]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[37:7] [data_stream_template] unknown field [failure_store]"
    }
  },
  "status": 400
}

Is the failure_store parameter correct under data_stream?

If you look at the copies under composable-logsdb there is a Jinja annotation that is selectively including the line

"data_stream": {
      "hidden": false,
      "allow_custom_routing": false
      {% if build_flavor != "serverless" %},
      "failure_store": false
      {% endif %}
    }

based on build_flavor. So, it is correct depending on build_flavor. When updating files please make sure that Jinja logic is preserved.

[achuguy]

Looks like failure_store is in the 8.16.0-SNAPSHOT and not in 8.15.x so I've adjusted my testing.

[gbanasiak]

I gave it a first pass. I may report more once I run the new track definition against a vanilla 8.15.1 cluster. I've labelled the PR with backport-to-8.15 to trigger automatic backport once this one is merged.

[salvatore-campagna]

LGTM after applying changes commented by @gbanasiak

[gbanasiak]

@achuguy I've added a92162e to avoid collision between packages required by Rally, and packages installed by the track.

[gbanasiak]

@salvatore-campagna

Looks like failure_store is in the 8.16.0-SNAPSHOT and not in 8.15.x so I've adjusted my testing.

We can't backport it to 8.15 with failure_store, this will need adjustment.

[pquentin]

To be clear, this is not a 8.16 vs 8.15 thing. failure_store is only enabled in SNAPSHOT builds and should go away completely at some point.

[gbanasiak]

Now with logs-endpoint composable templates being up-to-date we can simplify further by removing composable-logsdb directory. I'm also decoupling index mode setting from exclusions for benchmark scenarios with Kibana by introducing additional logs_endpoint_from_kibana track parameter (otherwise it's not possible to run in LogsDB mode standalone, without Kibana). See 9e1089b.

[gbanasiak]

2bafd52 moves the track closer to being fully supported in serverless (both standard user and operator). What remains to be done on that front (outside of this PR) is similar adjustment of Beats composable templates and addition of serverless IT tests (here).

[salvatore-campagna]

To be clear, this is not a 8.16 vs 8.15 thing. failure_store is only enabled in SNAPSHOT builds and should go away completely at some point.

Thi si because it is using a feature flag right? That at some point will be removed.

[salvatore-campagna]

Thanks @gbanasiak and @achuguy for fixing this!