Enable logsdb index mode in security track

[salvatore-campagna]

This PR changes the security track so that we can enable LogsDB
in index templates. Note that the failure store is only available in serverless
so we gate its usage excluding it in case the deployment is not serverless.

For LogsDB testing we rely on Kibana to install all other component/composable
templates. This is to make sure we need limited changes to the Rally track.

While testing this new configuration we discovered that installation of (component)
templates done by Kibana is Serverless only happens when a user interacts with it.
This means (component) templates are not installed and the elastic/security track
execution fails as a result of using (component) templates that do not exist.

This needs to be backported to branch 8.15.

[salvatore-campagna]

I am testing this using esbench both for stateful and serverless with a smaller dataset.

[salvatore-campagna]

@gareth-ellis @gbanasiak Serverless worked, can we enable at lest the serverless nightly?

[salvatore-campagna]

Stateful failed with the following error

esrally.exceptions.RallyError: Cannot run task [create-all-composable-templates]: Request returned an error. Error type: api, Description: invalid_index_template_exception ({'error': {'root_cause': [{'type': 'invalid_index_template_exception', 'reason': 'index_template [logs-endpoint.events.file] invalid, cause [index template [logs-endpoint.events.file] specifies component templates [logs-endpoint.events.file@package] that do not exist]'}], 'type': 'invalid_index_template_exception', 'reason': 'index_template [logs-endpoint.events.file] invalid, cause [index template [logs-endpoint.events.file] specifies component templates [logs-endpoint.events.file@package] that do not exist]'}, 'status': 400}), HTTP Status: 400

which is surprising because it means Kibana did not install the component template.

[salvatore-campagna]

After the latest commit 6bfdd20 I tested both serverless and stateful and I see both complete successfully. For serverless I had to exclude yet another component template which is not installed by Kibana apparently.

[gbanasiak]

I don't understand why 6bfdd20 was needed. The earlier custom component template exclusion is required due to elastic/rally#1855, but logs-endpoint.events.file@package should be present.

Stateful cluster after Kibana installed latest endpoint package:

GET _cat/component_templates/*endpoint*?v&s=name
name                                         version alias_count mapping_count settings_count metadata_count included_in
.logs-endpoint.action.responses@package              0           14            3              3              [.logs-endpoint.action.responses]
.logs-endpoint.actions@package                       0           17            3              3              [.logs-endpoint.actions]
.logs-endpoint.diagnostic.collection@package         0           5             3              3              [.logs-endpoint.diagnostic.collection]
.logs-endpoint.heartbeat@package                     0           7             5              3              [.logs-endpoint.heartbeat]
logs-endpoint.alerts@package                         0           29            4              3              [logs-endpoint.alerts]
logs-endpoint.events.api@package                     0           12            3              3              [logs-endpoint.events.api]
logs-endpoint.events.file@package                    0           16            3              3              [logs-endpoint.events.file]
logs-endpoint.events.library@package                 0           16            3              3              [logs-endpoint.events.library]
logs-endpoint.events.network@package                 0           16            3              3              [logs-endpoint.events.network]
logs-endpoint.events.process@package                 0           17            3              3              [logs-endpoint.events.process]
logs-endpoint.events.registry@package                0           15            3              3              [logs-endpoint.events.registry]
logs-endpoint.events.security@package                0           13            3              3              [logs-endpoint.events.security]
metrics-endpoint.metadata@package                    0           9             5              3              [metrics-endpoint.metadata]
metrics-endpoint.metrics@package                     0           9             3              3              [metrics-endpoint.metrics]
metrics-endpoint.policy@package                      0           9             3              3              [metrics-endpoint.policy]

Serverless security project:

GET _cat/component_templates/*endpoint*?v&s=name
name                                         version alias_count mapping_count settings_count metadata_count included_in
.logs-endpoint.action.responses@package              0           14            2              3              [.logs-endpoint.action.responses]
.logs-endpoint.actions@package                       0           17            2              3              [.logs-endpoint.actions]
.logs-endpoint.diagnostic.collection@package         0           5             2              3              [.logs-endpoint.diagnostic.collection]
.logs-endpoint.heartbeat@package                     0           7             4              3              [.logs-endpoint.heartbeat]
logs-endpoint.alerts@package                         0           29            3              3              [logs-endpoint.alerts]
logs-endpoint.events.api@package                     0           12            2              3              [logs-endpoint.events.api]
logs-endpoint.events.file@package                    0           16            2              3              [logs-endpoint.events.file]
logs-endpoint.events.library@package                 0           16            2              3              [logs-endpoint.events.library]
logs-endpoint.events.network@package                 0           16            2              3              [logs-endpoint.events.network]
logs-endpoint.events.process@package                 0           17            2              3              [logs-endpoint.events.process]
logs-endpoint.events.registry@package                0           15            2              3              [logs-endpoint.events.registry]
logs-endpoint.events.security@package                0           13            2              3              [logs-endpoint.events.security]
metrics-endpoint.metadata@package                    0           9             4              3              [metrics-endpoint.metadata]
metrics-endpoint.metrics@package                     0           9             2              3              [metrics-endpoint.metrics]
metrics-endpoint.policy@package                      0           9             2              3              [metrics-endpoint.policy]

logs-endpoint.events.file@package is present in both.

[gbanasiak]

@salvatore-campagna Can we modify track README to include the information about index_mode and a note that logsdb is for internal testing only?

[gbanasiak]

Thank you for all the iterations. LGTM.