Opentelemetry Trace graph becomes disconnected after creating Grpc Call

[GedasFX]

What version of gRPC and what language are you using?

Grpc.AspNetCore 2.65.0

What operating system (Linux, Windows,...) and version?

Windows 10 22H2

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

8.0.400

What did you do?

I instrumented an application with open telemetry for tracing. The tracing tree is disjointed because one activity is being created and started regardless if there are listeners.

What did you expect to see?

A trace span not to be collected (even started) if there are no listeners.

What did you see instead?

The graph of code should show the following:

1. A request came to the service
2. Service made a request to another service (for simplicity it was to itself)
3. Service did the same as step 1, but on different endpoint.

In the image you can see 2 root spans. They are not unrelated, and should be part of the tree. This is the instrumentation code:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(o => o.AddService("my_service").Build())
    .WithTracing(o => o
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter()
        .AddOtlpExporter(o => o.Endpoint = new Uri("http://localhost:4317")));

As a debugging step I tried instrumenting everything, added .AddSource("*"). This revealed the culprit:

The graph now is now intact, however it has additional instrumentation that is not really needed.

After digging around, I suspect this is the culprit code:

grpc-dotnet/src/Grpc.Net.Client/Internal/GrpcCall.cs

Line 826 in e9cc7e1

activity = GrpcDiagnostics.ActivitySource.CreateActivity(GrpcDiagnostics.ActivityName, ActivityKind.Client);

To my understanding of OpenTelemetry, if there are no listeners to the source, then it is not created, and because of that the trace tree is intact, but here since we force create and start an activity, the parent ids get messed up, and so does the trace tree.

Anything else we should know about your project / environment?

Repro code: https://github.com/GedasFX/opentelemetry-dotnet-grpc-failure