dotnet-trace on containers

[Jongy]

Background and Motivation

This is for Linux, when using dotnet-trace in PID connect mode, e.g dotnet-trace collect --process-id XXX ....

dotnet-trace attempts to connect to the transport pipe in /tmp of the mount namespace it's currently running in. If the target PID runs in a container (or if dotnet-trace itself runs in a container), then its mount namespace is different than the mount namespace of dotnet-trace, and we'll not find its pipe.

This is raised in #810 and the proposed solution (at least from my skimming) is to mount the /tmp such that it's accessible on the host.

Proposed Feature

Another solution that I'm proposing is to access the /tmp of the container, via /proc/pid/root/tmp, in case the target process runs in a different mount namespace (easily checked by comparing the /proc/pid/ns/mnt file of dotnet-trace to the one of the process).

We're using this "trick" in our gProfiler (see usage here), by changing the TMPDIR variable which is respected by dotnet-trace, and it's proved itself to work, so I figured we could place this logic in upstream and have it fixed for everyone :)

I searched through the code a bit and found PAL_GetTransportName which appears to be the relevant function for this change. It also seems to handle the same concept for MacOS (using GetApplicationContainerFolder). I figure that'd be the place to put the matching Linux support.

If you think this could be a good addition, I'm happy to implement this change.

Usage Examples

Profiling/tracing a process running in a container.

[tommcdon]

@Jongy thank you for writing up the proposal and offer to implement improving dotnet-trace scenarios for correctly handling container namespaces for the mounted directories. Adding @noahfalk for thoughts.

[noahfalk]

If I am following correctly the goal is to adjust where dotnet-trace searches for socket based on the filesystem mount of the target process being traced? That sounds useful to me and we'd be glad to get a PR 👍. Thanks!

dotnet-trace (and all the other dotnet-*) tools use this code to enumerate available processes:

diagnostics/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs

Line 310 in 32e474a

foreach (var port in Directory.GetFiles(PidIpcEndpoint.IpcRootPath))

and this code to connect to get the address for a specific process by PID:

diagnostics/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs

Line 300 in 3128b54

defaultAddress = Directory.GetFiles(IpcRootPath, $"dotnet-diagnostic-{pid}-*-socket") // Try best match.

The PAL_GetTransportName function is slightly different, that one gets used by the debugger to determine the path to the debugging pipe. For somewhat arbitrary historical reasons the debugger pipe is distinct from the diagnostics pipe that all other tools use.

[Jongy]

If I am following correctly the goal is to adjust where dotnet-trace searches for socket based on the filesystem mount of the target process being traced? That sounds useful to me and we'd be glad to get a PR +1. Thanks!

Yes.

and this code to connect to get the address for a specific process by PID:

Oh wow, I was way off in my search 😅

Okay, that's all the information I need to know. Might take me a while to reach it but I'll try making the modification and if I prove it works, I will send a PR :) Thanks for the replies guys!

[Jongy]

I pushed a PoC PR in #3539 , would love your comments :)