-
Notifications
You must be signed in to change notification settings - Fork 480
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: SPM monitor tab does not show Latency data #2088
Comments
SummaryHello, I'm experiencing an issue with the Jaeger Service Performance Monitor (SPM) dashboard where the latency data consistently shows "No Data" for all services. Upon inspecting the query in my web browser, I found that the metrics for "service_latencies" were empty. Here's the information from the query inspection:
I checked the Jaeger documentation and found that the relevant metric for this dashboard is "latency_bucket," which is deprecated on the OpenTelemetry side. Application Versionjaeger version: 1.54 |
@chkp-talron @MinaMohammadi Am experiencing the same issue. have you found any fixes for this issue?? @ jaeger team @yurishkuro : Can you please look into this issue if not already. |
The documentation has quite detailed troubleshooting steps https://www.jaegertracing.io/docs/latest/spm/#troubleshooting. Which of them did you try and what were the results? |
@yurishkuro Initially we did not see any metrics on the SPM. But after the above inclusion of ENV, the error rate and request rate started to appear, while the latency metrics were not appearing. Please do let us know if you require any other information, will be glad if you resolve our issue here. |
@yurishkuro any further help on this? because of this issue we had to rollback to older opentelemetry version |
What happened?
Installed jaeger deployment on k8s cluster, jaeger works properly. I was able to enable the monitor tab by setting various flags in jaeger deployment yamls, but I'm able to see only Request rate data for all of my services , and Error rate data for a single service out of 15 services. but Latency shows "No Data" and the top of the page reads: "No data yet! Please see these instructions on how to set up your span metrics."
i assume that if at least one service show errors data, than the other services does not have errors (the service which shows error if kind of a gateway into our cluster, so it might be reasonable it shows errors and the other dont.)
however, latency is empty all the time for all services. i cant tell if this is due to how metrics are sent to prom, there are tone sof attributes on each on of the metrics, so maybe aggregation needs to be on only a few attributes to be able to show latency data?
I'm sending app metrics using otel SDK,
Collector is configured with spanmetrcis connector
Steps to reproduce
Expected behavior
SPM monitor should show latency data
Relevant log output
No response
Screenshot
Additional context
No response
Jaeger backend version
Helm Chart: jaeger-0.73.1 Jaeger version: 1.51.0
SDK
Otel nodejs SDK:
"@opentelemetry/api": "^1.4.1",
"@opentelemetry/core": "^1.15.2",
"@opentelemetry/exporter-metrics-otlp-http": "^0.41.2",
"@opentelemetry/exporter-trace-otlp-http": "^0.41.2",
"@opentelemetry/sdk-metrics": "^1.15.2",
"@opentelemetry/sdk-trace-base": "^1.15.2",
"@opentelemetry/sdk-trace-node": "^1.15.2"
Pipeline
OTel SDK -> Otel Collector -> jaeger Collector -> ES
Stogage backend
ES 7
Operating system
Linux
Deployment model
EKS
Deployment configs
The text was updated successfully, but these errors were encountered: