You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe
When we use Opensearch stack to ingest logs, it's normal to use data stream index.
More over, It's normal on this use case to have hot / warm / delete architecture. We keep 24h of logs on hot node, then we keep them in long time (30d) on warm node.
Because of logs are not always searched on warm node, we shrink index to avoid ta have to many shard on warm nodes. So on ISM (index state management) policy we have shrink step on warm phase. But this step not working as expected with data stream index. When it create new index (ths shrunken index), it break the lineage with data stream index. The new index is no more part of data stream index. And it break all the next step of ISM because of the new index is outdoor of the current policy.
Describe the solution you'd like
We expect that this ISM policy work out of the box.
As human understanding, we should it work like this:
Hot phase when new data stream index is created or new rollover index
When index exist during more than 1 day or one primary shard is biggest than 50Gb, we create new index and for previous index, we go on warm phase (this one work fine)
Warm phase after rollover
It move all shard / replica of previous index on warm nodes (this one work fine)
It change de index priority of previous index to 50 (this one work fine)
It shrink the previous index to have 2 primary shard from 4. We expect the new index that reflect the shrunken action continue to be a part of datastream index. We also expect that all next step will be applied on shrunken index. And the source of shrunken index will be automatically deleted. (this one not work)
it force merge the shrunken index
After 30d, the shrunken index need to go on delete phase
Delete phase
It delete the shrinken index
It can be usefull to orchestrate the steps from phase to the best order. Maybee the best order is not the order provided by user.
Maybee it's better to start with force merge and finish by allocation ... Or maybee force merge need to be run after shrink index and allocate it on the right node ?
Related component
Other
Describe alternatives you've considered
No response
Additional context
I think shrink step must be work like rollover step with data stream index. After look a little bit the code, I think the problem is not on ISM plugin, but on Opensearch core. Especially on server/src/main/java/org/opensearch/action/admin/indices/shrink package.
The text was updated successfully, but these errors were encountered:
Thanks for filing. This looks like a reasonable request. We will need the ability to add indexes to a data stream as a prerequisite (see opensearch-project/OpenSearch#8271).
Is your feature request related to a problem? Please describe
When we use Opensearch stack to ingest logs, it's normal to use data stream index.
More over, It's normal on this use case to have hot / warm / delete architecture. We keep 24h of logs on hot node, then we keep them in long time (30d) on warm node.
Because of logs are not always searched on warm node, we shrink index to avoid ta have to many shard on warm nodes. So on ISM (index state management) policy we have shrink step on warm phase. But this step not working as expected with data stream index. When it create new index (ths shrunken index), it break the lineage with data stream index. The new index is no more part of data stream index. And it break all the next step of ISM because of the new index is outdoor of the current policy.
Describe the solution you'd like
We expect that this ISM policy work out of the box.
As human understanding, we should it work like this:
Related component
Other
Describe alternatives you've considered
No response
Additional context
I think shrink step must be work like rollover step with data stream index. After look a little bit the code, I think the problem is not on ISM plugin, but on Opensearch core. Especially on server/src/main/java/org/opensearch/action/admin/indices/shrink package.
The text was updated successfully, but these errors were encountered: