[Feature]: Support elasticsearch data stream

[pengweiqhca]

Requirement

https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

Problem

I has created a data stream index template, and I tried --es.use-aliases=true but not work.

Proposal

No response

Open questions

No response

[jack78901]

Is this because the client library used for connecting to Elasticsearch is outdated? Namely, I see in the logs of version 1.54 of the collector that the client is olivere/[email protected]. This client is from July 27th, 2021. The client is also not being updated anymore, it would appear. Namely, it was last updated to version 7.0.32 on Mar 19th, 2022. This is almost two years ago now. Elastic has its own go library now: https://github.com/elastic/go-elasticsearch.

[tronda]

I see that the OpenSearch project has the same feature-request.

[JaredTan95]

I am currently working on this feature, but I prefer to implement it in jaeger v2

[yurishkuro]

@JaredTan95 +2 for v2, but what specifically is the difference? Our storage implementations are currently identical in v1 and v2 (with the exception of configuration)

[JaredTan95]

@JaredTan95 +2 for v2, but what specifically is the difference? Our storage implementations are currently identical in v1 and v2 (with the exception of configuration)

Currently, if we want to use esILM policy, we need to create aliases and we need to create the xxx-00001 initial index before jaeger-colelctor writes data. This increases the difficulty of operation and maintenance. ES data stream shields users from the above problems. We no longer need to do some bootstrap works, and can directly write data to es through data stream. Scrolling of indexes and switching of aliases are managed within the data stream its-self.

[yurishkuro]

1. that doesn't really address my question - all ES-specific details are hidden behind es.Factory, which is used by both Jaeger v1 and v2, so I don't see a difference
2. Data streams also need to be set up once, just like index mappings. We want to do that on collector start-up, with some coordination between multiple collector instances (if creation operation are idempotent then it's less of an issue)