Large-transfer high-speed stream and datagram client/servers [Why am I slower than TCP?]

[ertza]

Hi, I am working on creating a high-speed client/server using msquic, I don't need encryption and hence I have disabled that with QUIC_PARAM_CONN_DISABLE_1RTT_ENCRYPTION in my settings. I am sending 5120000 bytes of data as a test and sending through Stream and Datagram one after the other.

The TCP program I wrote takes 10x less time (on a public cloud network) to transfer the data compared to my msquic approach despite disabling encryption.

Also, my Datagram approach takes even longer than Stream. For Stream I am giving the complete buffer to the API at once like this:

int message_size = 51200000;
int SendBufferLength = message_size;
SendBufferRaw = (uint8_t*)malloc(sizeof(QUIC_BUFFER) + SendBufferLength);
SendBuffer = (QUIC_BUFFER*)SendBufferRaw;
SendBuffer->Buffer = SendBufferRaw + sizeof(QUIC_BUFFER);
SendBuffer->Length = SendBufferLength;
StreamSend(Stream, SendBuffer, 1, QUIC_SEND_FLAG_FIN, SendBuffer);

Whereas for Datagram I cannot give more than around 1100 as buffer length for one send, so I set it to 1024 and then do a loop to send all the buffers one by one:

int udpSendBufferLength = 1024;
int total_buffers = message_size/udpSendBufferLength;
QUIC_BUFFER SendBuffers[total_buffers];
while (i<total_buffers) {
  MsQuic->DatagramSend(Connection, &SendBuffers[i++], 1, QUIC_SEND_FLAG_NONE, NULL)
}

What is the optimal approach in this case to transfer the data fast, I am not concerned about security or reliability and only looking for the optimal way to transfer the data both in Stream and Datagram scenarios (I expect the Datagram to be much faster than Stream as well).

[nibanks]

Streams will be much faster than Datagrams. You might want to turn off send buffering for the connection, but that won't make more than a 10% difference most likely. But raw TCP is usually faster than UDP because of the availability of offloads (on send and recv). But ideally, we could grab some Datapath.Light logs and analyze them with WPA to see what's up. Also, what OS & version are you using?

[ertza]

Any intuition on why Streams are faster? In my understanding Datagram doesn't do retransmissions so shouldn't it be faster than Streams?

I agree about TCP offloads but I am able to get similar speeds with my standalone TCP and UDP servers (around 15ms compared to QUIC 150ms), I did have to increase the buffer in linux with net.core.wmem_max etc. and the UDP socket in C with setsockopts SO_SNDBUF. I send 65k bytes of data in every sendto call to this UDP socket and this allows to me get similar speeds as TCP. Is there any way to send bigger buffers in MSQUIC Datagram as well? I am only able to send around 1100 bytes and I saw there are some Settings for Streams but couldn't find any for Datagrams.

I'm on Ubuntu 20.04 but I am not familiar with Datapath.Light logs, is it supported for Linux? Or should I use something else to get the logs e.g. enabling logging with this QUIC_ENABLE_LOGGING=ON -D QUIC_LOGGING_TYPE=stdout

[nibanks]

Datagrams only allow you to send 1 packet's worth of content at a time, which just queues to the MsQuic worker thread (lots of context switches back and forth, as well as repeated synchronization). Streams allow you to queue once (or few times) and then the worker threads just spins building and sending all the packets out.

The logs are the best next step. You will need to use lttng instead of stdout logging mode. Logging on Linux is more expensive than on Windows too. It'd be interesting to know if you have the same problem on Windows.

[ertza]

Makes sense. Also, disabling send buffer didn't seem to help. Going to collect logs now, will update once I collect.

Thanks for the prompt replies.