Relatively high latency spikes when streaming messages to a client on the same machine

[jakupj]

I'm using grpc to stream messages from a service to a client on the same machine. I expect it to have pretty low latency, ie sub millisecond. For the most part it does, however I'm seeing very high spikes in latency.

I wrote a simple test client/server to test grpc steaming latency src here: https://github.com/jakupj/GrpcTesting

Latency is determined by timestamping the message at the service just before sending it the the client and then timestamping on the client. For the most part, ie the 99th percentile I'm seeing sub 1ms latency. However I do see extremely high spikes in latency ie 30ms and over.

I've tried all the suggestions in the performance docs, even setting the gc to SustainedLowLatency mode, with no effect. Tried it on two different machines, one a Windows 11 system and another Windows server 2022 with no difference. My suspicions are that it might be GC related.

I've also tried running the serverstreaming benchmarks in the grpc-dotnet repo, I'm seeing similar results of high max latency there too.

My requirements are to stream messages to clients on the same machine with peaks around 5,000 or so messages per second with sub millisecond consistently.

Is there something I'm missing, a configuration setting maybe, am I doing it correctly? Or should I be looking at another solution, maybe signalR instead?

Thanks