How to configure Msquic for maximum throughput

[honkstar1]

Hi,

We have a very network intense application that is currently using raw tcp. The solution dynamically opens more tcp sockets if latency is high (1 socket per 15ms ping) and I can saturate my network connection really well with this setup.

Right now I'm testing integrating msquic since I want the security and also hoping to be able to implement udp hole punching once switched over to msquic.

My problem though is that I can't get nearly as high throughput as the tcp solution and I suspect it might be related to how I'm using msquic... so this post is an ask for how to configure quic for maximum throughput.

Here's my setup.
Superfast host computer (Threadripper pro, 256gb ram) using 0.6% cpu when running our application
Very stable internet connection (3gbps symmetric fiber directly in to my house)
Latency at 80ms (I'm located on west coast NA using AWS us-east)

Tcp solution
Throughput peaks at 2.5gbps+ with 4 tcp sockets... much lower with only one socket. (I guess limitation of sliding window and ack. With lower latency I get same throughput using only 1 socket)

Msquic solution
Throughput peaks at ~800mbps

Right now I have no modifications to Msquic configuration and just use the setup from sample.c. I have tested creating multiple streams inside the same connection without seing any throughput differencies. I've tested 1 stream all the way up to 10 streams.

Are there settings I can play with to try to improve the throughput?

Would it help to open more connections? Since it is udp I would expect that to not really matter (there is no competition going on on my machine with other network intense activity)

Also, a side note is that I'm using OpenSSL because I couldn't get the schannel path to work (I will revisit this at some point). I see in your perf benchmarks that schannels are faster but I wouldn't expect it to "solve" my scenario.

Any advise would be appreciated :-)

Best regards,
Henrik

[anrossi]

The first thing that comes to mind is that the sample uses send buffering by default. If you can guarantee that you have at least 2 send buffers pending at all times, you can turn off send buffering and get more performance. We have multiple Gbps performance with a single MsQuic connection, so multiple connections wouldn't help here; they'd be competing with each other for bandwidth and end up with worse performance. Beyond that, we might need some performance traces to see what other bottlenecks there are. If you don't want to share them, you can use the instructions at https://github.com/microsoft/msquic/blob/main/src/plugins/trace/README.md to use the Windows Performance Analyzer (WPA) plugin to investigate performance bottlenecks. Let us know if you have more questions, or problems investigating this further.

[honkstar1]

Thanks for the response,

I forgot to add that the send pattern is calling send on the same thread multiple times with 256kb data. in the tcp implementation we round robin tcp socket for each send call. There are no recv or anything in between.

In my quic test I do the same but round robin quic streams inside the same connection.

On the receiving side I use QUIC_STREAM_EVENT_RECEIVE and leave the thread as quickly as possible so no contention there.

I will try to get a performance trace to you :-)

Thanks again,
Henrik

[nibanks]

We'll need traces to analyze with WPA. Feel free to look yourself, or share the traces with us. Thanks!

[honkstar1]

I was stepping through the msquic code and it seems like only time the sendinline works is if send is called from the msquic connection worker thread.

--- api.c ---
SendInline =
!Connection->Settings.SendBufferingEnabled &&
Connection->WorkerThreadID == CxPlatCurThreadID();

Our design is to do minimum blocking on recv thread and marshal data somewhere else where all the work is done and then those worker threads send the results over socket. (the worker threads do tons of work)

Is the idea with msquic that you do the work in the thread that gets the QUIC_STREAM_EVENT_RECEIVE call? I see msquic spawns 128 worker threads on my machine... and it looks like those are only to drain the command queue? That is a lot of thread if very little work is done on them? or is the encryption/decryption taking so much overhead?

I will get to the trace.. just stepping msquic code right now to see how well this integrates with our setup. If there are design decisions that needs to change because using msquic works differently than tcp sockets.

[honkstar1]

This article claims lower latency and better perf using multiple connections.

https://techcommunity.microsoft.com/t5/networking-blog/balance-performance-in-msquic-and-xdp/ba-p/3627665

I will test multiple connections to see if it affects outcome.

I will also test creating a new stream for every 256kb message using start/fin flags.

[nibanks]

Yes, MsQuic doesn't send anything out on the app's thread. You can get that inline send flush if you happen to be calling send on a different MsQuic callback. But even this doesn't actually send packets on the wire. It queues things up for later. We use dedicated threads to drive execution internally. This greatly simplifies the logic (mostly no locking), and usually provides plenty of performance for most app needs (even servers). If you really need more, yes, you can create multiple parallel connections.

a new stream for every 256kb message

One thing to watch out here is flow control. If you expect to send such large chunks of data on each new stream, make sure the peer configured the StreamRecvWindowDefault to be at least this amount, otherwise it will take multiple round trips for the data to be exchanged.