Will the crash of jaeger components put my service in danger ?

[chenxinlong]

Hi, I'm wondering :

Q1 : If the crash of jaeger components(like jaeger-agent, jaeger-collector or the backend storage) will put my service in danger?

Q2 : If the jaeger-collector crashed, will jaeger-agent drop spans directly or store spans for a while ? If jaeger-agent store spans then how much spans will it stores ?

[albertteoh]

@chenxinlong

Q1: The jaeger Go client, which I'm most familiar with, sends (with some basic connection retry logic in the underlying UDP client), accumulates failure statistics and reports these stats to the agent on a subsequent successful send to the agent. In the event of a crash of jaeger-agent, jaeger-collector or backing store, the jaeger Go client will not crash, nor accumulate spans in memory (and retry at a later time).

I don't have a good understanding of other language SDKs but would think they exhibit similar behaviour.

Q2: If jaeger-collector crashes, the jaeger-agent will drop spans and does not retry. Dropped spans are logged by the agent and dropped span counts are reported against the jaeger_agent_reporter_batches_failures_total metric.

[chenxinlong]

Thanks, very helpful.

[jpkrohling]

To complement @albertteoh's answer: no client should ever make your application fail. If they do, we consider it a bug. About the agent: when the collector isn't available, the gRPC client will retry the failed connections, so, the agent will keep some batches in memory until they eventually fail.

[chenxinlong]

Thanks a lot.

the agent will keep some batches in memory until they eventually fail.

Since the jaeger-agent is deployed with application, will the memory usage of jaeger-agent grows to a high level during the period when collctor is unavailable ?

[jpkrohling]

Since the jaeger-agent is deployed with application

You probably mean in a sidecar fashion, but in all other cases, the agent isn't tied to your application in any way.

will the memory usage of jaeger-agent grows to a high level during the period when collctor is unavailable ?

I don't remember now from the top of my head, but a moderate increase is indeed expected but work should be placed in queues, which would drop work when the queue is full. In other words, the agent should not cause your application to stop working in any way either.

[chenxinlong]

You probably mean in a sidecar fashion

Yes

In other words, the agent should not cause your application to stop working in any way either.

Thanks