Jaeger elasticsearch index cleaner feature not working #3755

kavndu · 2022-06-15T11:57:58Z

kavndu
Jun 15, 2022

Iam working on enabling esIndexCleaner feature by using jaeger official helm chart. Here is the changes that i did in values.yaml file (helm-charts/charts/jaeger/values.yaml) after cloning the repository.

Change the default storage service to elasticsearch instead of cassandra.
Enabled Jaeger query and the hotrod application as LoadBalancer services.
In the esIndexCleaner section i have added below configurations to enable index cleaner and delete the elasticsearch indices after a one day. I have attached a screenshot below. Left hand side is the default helm chart values.yaml configurations and there is updated configurations of esIndexCleaner section in the right hand side.

After upgrading helm charts and after 26/27 hours i checked the indices in the elasticsearch pod and also jaeger spans from the Jaeger query dashboard. But indices had not deleted. Jaeger old spans was there in the query dashboard also. Is there any other configurations should i add? Appreciate if you can suggest a solution for this.

Here i have attached the output of the kubectl get pods and kubectl get cronjob command outputs

#######@######################:~/#########/helm-charts/charts/jaeger$ kubectl get pods
NAME                                       READY   STATUS      RESTARTS   AGE
elasticsearch-master-0                     1/1     Running     0          44h
elasticsearch-master-1                     1/1     Running     0          32h
elasticsearch-master-2                     1/1     Running     0          44h
jaeger-agent-#####                         1/1     Running     0          44h
jaeger-agent-#####                         1/1     Running     0          44h
jaeger-agent-#####                         1/1     Running     0          44h
jaeger-collector-cbcdc758b-#####           1/1     Running     0          32h
jaeger-es-index-cleaner-##########-#####   0/1     Completed   0          145m
jaeger-es-index-cleaner-##########-#####   0/1     Completed   0          85m
jaeger-es-index-cleaner-##########-#####   0/1     Completed   0          25m
jaeger-hotrod-##########-#####             1/1     Running     0          32h
jaeger-query-##########-#####              2/2     Running     0          32h



#######@######################:~/#########/helm-charts/charts/jaeger$ kubectl get cronjob
NAME                      SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
jaeger-es-index-cleaner   0 * * * *   False     0        26m             43h

Here is the kubectl describe po elasticsearch-master-0 command output.

delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                             elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:          elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
      discovery.seed_hosts:                  elasticsearch-master-headless
      cluster.name:                          elasticsearch
      network.host:                          0.0.0.0
      cluster.deprecation_indexing.enabled:  false
      node.data:                             true
      node.ingest:                           true
      node.master:                           true
      node.ml:                               true
      node.remote_cluster_client:            true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-##### (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  default-token-#####
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-#####
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

pavolloffay · 2022-06-20T16:54:03Z

pavolloffay
Jun 20, 2022
Maintainer

@kavndu could you please provide output from the es-index-cleaner job(s)?

1 reply

kavndu Jul 1, 2022
Author

Hi @pavolloffay

Thank you for your response, Here i have attached es-index-cleaner logs.

{"level":"info","ts":1656676927.7005563,"caller":"./main.go:86","msg":"Indices before this date will be deleted","date":"2022-07-01T00:00:00Z"}
{"level":"info","ts":1656676927.7006402,"caller":"./main.go:95","msg":"Queried indices","indices":[{"Index":"jaeger-span-2022-06-30","CreationTime":"2022-06-30T00:00:02.686Z","Aliases":{}},{"Index":"jaeger-span-2022-07-01","CreationTime":"2022-07-01T00:00:01.818Z","Aliases":{}},{"Index":"jaeger-service-2022-06-28","CreationTime":"2022-06-28T10:00:32.688Z","Aliases":{}},{"Index":"jaeger-service-2022-06-29","CreationTime":"2022-06-29T03:22:52.683Z","Aliases":{}},{"Index":"jaeger-service-2022-06-30","CreationTime":"2022-06-30T05:37:00.684Z","Aliases":{}},{"Index":"jaeger-service-2022-07-01","CreationTime":"2022-07-01T10:41:57.811Z","Aliases":{}},{"Index":"jaeger-span-2022-06-28","CreationTime":"2022-06-28T10:00:32.864Z","Aliases":{}},{"Index":"jaeger-span-2022-06-29","CreationTime":"2022-06-29T00:00:02.686Z","Aliases":{}}]}
{"level":"info","ts":1656676927.7008142,"caller":"./main.go:99","msg":"No indices to delete"}

Is there any other required configurations need to be added in the values.yaml?

pavolloffay · 2022-07-01T13:52:17Z

pavolloffay
Jul 1, 2022
Maintainer

@kavndu what Jaeger version are you runnning?

Could you please paste output of kubectl describe from the es-cleaner cron job?

1 reply

kavndu Jul 1, 2022
Author

Here is the kubectl describe command output

kubectl describe po jaeger-es-index-cleaner-xxxxxxxxxx-xxxxx

Name:         jaeger-es-index-cleaner-1656676920-zt7v8
Namespace:    default
Priority:     0
Node:         gke-cluster-1-default-pool-805f95ab-eet0/10.128.15.208
Start Time:   Fri, 01 Jul 2022 17:32:07 +0530
Labels:       app.kubernetes.io/component=es-index-cleaner
              app.kubernetes.io/instance=jaeger
              app.kubernetes.io/name=jaeger
              controller-uid=c83d3053-906c-4fff-af01-5ef9fff5a00f
              job-name=jaeger-es-index-cleaner-1656676920
Annotations:  <none>
Status:       Succeeded
IP:           10.4.2.32
IPs:
  IP:           10.4.2.32
Controlled By:  Job/jaeger-es-index-cleaner-1656676920
Containers:
  jaeger-es-index-cleaner:
    Container ID:  containerd://63ff4ab1a06565f1ef2d06c9020df521999b1cc5c5f6e3c2a08405ea18c6a68f
    Image:         jaegertracing/jaeger-es-index-cleaner:1.30.0
    Image ID:      docker.io/jaegertracing/jaeger-es-index-cleaner@sha256:6910a7f5c42022b38d333c983a5305e49ce88823fbd84dcf6923ea1f3244e065
    Port:          <none>
    Host Port:     <none>
    Args:
      1
      http://elasticsearch-master:9200
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 01 Jul 2022 17:32:07 +0530
      Finished:     Fri, 01 Jul 2022 17:32:07 +0530
    Ready:          False
    Restart Count:  0
    Environment:
      ROLLOVER:                 true
      ES_TLS_ENABLED:           true
      ES_TLS_SKIP_HOST_VERIFY:  true
      ES_SERVER_URLS:           http://elasticsearch-master:9200
      ES_USERNAME:              elastic
      ES_PASSWORD:              <set to the key 'password' in secret 'jaeger-elasticsearch'>  Optional: false
    Mounts:                     <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:             <none>

Jaeger agent docker version - jaegertracing/jaeger-agent:1.30.0
Jaeger collector docker version - jaegertracing/jaeger-collector:1.30.0
Jaeger query docker version - jaegertracing/jaeger-query:1.30.0

pavolloffay · 2022-07-01T14:48:27Z

pavolloffay
Jul 1, 2022
Maintainer

The configuration seems to be mismatched. The output from logs show that jaeger-collector is creating daily indices but but the index cleaner is configured to use rollover ROLLOVER: true. The rollover uses a different set of indices.

{"level":"info","ts":1656676927.7006402,"caller":"./main.go:95","msg":"Queried indices","indices":[{"Index":"jaeger-span-2022-06-30","CreationTime":"2022-06-30T00:00:02.686Z","Aliases":{}},{"Index":"jaeger-span-2022-07-01","CreationTime":"2022-07-01T00:00:01.818Z","Aliases":{}},{"Index":"jaeger-service-2022-06-28","CreationTime":"2022-06-28T10:00:32.688Z","Aliases":{}},{"Index":"jaeger-service-2022-06-29","CreationTime":"2022-06-29T03:22:52.683Z","Aliases":{}},{"Index":"jaeger-service-2022-06-30","CreationTime":"2022-06-30T05:37:00.684Z","Aliases":{}},{"Index":"jaeger-service-2022-07-01","CreationTime":"2022-07-01T10:41:57.811Z","Aliases":{}},{"Index":"jaeger-span-2022-06-28","CreationTime":"2022-06-28T10:00:32.864Z","Aliases":{}},{"Index":"jaeger-span-2022-06-29","CreationTime":"2022-06-29T00:00:02.686Z","Aliases":{}}]}

1 reply

kavndu Jul 1, 2022
Author

Now it's working. I disabled rollover for the index cleaner. Thank you @pavolloffay

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jaeger - Distributed Tracing Platform

Jaeger elasticsearch index cleaner feature not working #3755

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Jaeger - Distributed Tracing Platform

Jaeger elasticsearch index cleaner feature not working #3755

kavndu Jun 15, 2022

Replies: 3 comments · 3 replies

pavolloffay Jun 20, 2022 Maintainer

kavndu Jul 1, 2022 Author

pavolloffay Jul 1, 2022 Maintainer

kavndu Jul 1, 2022 Author

pavolloffay Jul 1, 2022 Maintainer

kavndu Jul 1, 2022 Author

kavndu
Jun 15, 2022

Replies: 3 comments 3 replies

pavolloffay
Jun 20, 2022
Maintainer

kavndu Jul 1, 2022
Author

pavolloffay
Jul 1, 2022
Maintainer

kavndu Jul 1, 2022
Author

pavolloffay
Jul 1, 2022
Maintainer

kavndu Jul 1, 2022
Author