[BUG]The new index after rollover is not managed by ism policy

[lliu0947]

Describe the bug
I create a data stream and add ism policy to the data stream, but after index rollover, the new index is not managed by ism policy。

Steps to reproduce the behavior:

1. create ism policy
   PUT _plugins/_ism/policies/test-ism-api
   {
   "policy": {
   "description": "an example of the ism policy",
   "default_state": "hot",
   "states": [
   {
   "name": "hot",
   "actions": [
   {
   "rollover": {
   "min_doc_count": 5
   }
   }
   ],
   "transitions": [
   {
   "state_name": "warm"
   }
   ]
   },
   {
   "name": "warm",
   "actions": [{
   "replica_count" : {
   "number_of_replicas" : 1
   }
   }
   ],
   "transitions": [
   {
   "state_name": "delete",
   "conditions": {
   "min_index_age": "5m"
   }
   }
   ]
   },
   {
   "name": "delete",
   "actions": [
   {
   "delete": {}
   }
   ],
   "transitions": []
   }
   ]
   }
   }

2. create index template
   PUT _index_template/logs-template-nginx
   {
   "index_patterns": "logs-nginx",
   "data_stream": {
   "timestamp_field": {
   "name": "request_time"
   }
   },
   "template": {
   "aliases": {
   "my_logs": {}
   },
   "settings": {
   "number_of_shards": 1,
   "number_of_replicas": 0
   }
   }
   }

3. create data stream
   PUT _data_stream/logs-nginx

4. get data stream information
   {
   "data_streams" : [
   {
   "name" : "logs-nginx",
   "timestamp_field" : {
   "name" : "request_time"
   },
   "indices" : [
   {
   "index_name" : ".ds-logs-nginx-000001",
   "index_uuid" : "f7gTvToLTkqKWN1L1Bc9ug"
   }
   ],
   "generation" : 1,
   "status" : "GREEN",
   "template" : "logs-template-nginx"
   }
   ]
   }

5. manage data streams in OpenSearch Dashboards
   open OpenSearch Dashboards, choose Index Management, select Indices or Policy managed indices. Follow the steps in the image below。
   

After index rollover, the new index is not managed by ism policy

in kibana, only one or more indexes in the data stream can be selected to associate the ism policy。Instead of associating the entire data stream with the ism policy, when rollover, you need to manually add the ism policy to the new index

Expected behavior
after index rollover，the new index is managed by ism policy

[dblock]

@lliu0947 did you solve your problem? how?

[lliu0947]

@lliu0947 did you solve your problem? how?

yeah，i have solved it

[dblock]

@lliu0947 how did you solve it?

[Promethesus]

What was the solution here? Has anyone found a fix for this?

[FriendlyTroll]

Sorry for necroposting but I was struggling with this and just want to leave some info if/when someone else also stumbles upon this via google.

The way I understand it, you need to create a policy that references the name of your datastream(s). So if the datastream is logstash-nginx-access-logs-srv-01 in a policy set the index pattern like logstash-nginx-access-logs-srv*. I also set a priority higher than 0.

Now in the index template that is used by your datastream I also added the backing index regex (in the Advanced settings) like so
"index.plugins.index_state_management.rollover_alias": ".ds-logstash-nginx-*". Change this depending on the name of the backing indexes.

Also the docs mention that

“The ISM policy is applied to the backing indexes at the time of their creation. When you associate a policy to a data stream, it only affects the future backing indexes of that data stream.”

meaning you will need to manually rollever the indexes the first time to apply the policy on future indexes. To do this select the datastreams then Actions → Roll over. After that, the policy should be autoapplied on future indexes.

[dblock]

@FriendlyTroll thank you! If you think the docs can be improved please do contribute

For anyone reading this, we'd also love some help on opensearch-project/opensearch-api-specification#224 (we plan to generate docs from there).