Set SO_REUSEADDR on epoll tcp listener sockets

[ThadHouse]

Description

Unix is a bit more strict about TIME_WAIT state, and actually puts any sockets that have had a valid accept() called on them into the TIME_WAIT state.

This makes writing a listener app difficult, as if that ever crashes the bind() will fail for the next few minutes.

Pretty much all other TCP libraries set SO_REUSEADDR (Including libuv, which is what our app has used before). Libuv sets it on all TCP sockets, but its generally less required on client sockets, as they rarely actually specify a local port.

Describe the purpose of and changes within this Pull Request.

Testing

Not really possible to test with our current setup, as a crash is the only way to get a listener socket into TIME_WAIT. A properly shutdown socket doesn't stay in TIME_WAIT.

Documentation

N/A

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.67%. Comparing base (733da8d) to head (21d8366).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4544      +/-   ##
==========================================
+ Coverage   86.37%   86.67%   +0.29%     
==========================================
  Files          56       56              
  Lines       15537    17354    +1817     
==========================================
+ Hits        13420    15041    +1621     
- Misses       2117     2313     +196     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[csujedihy]

I am a bit confused. The listener socket itself can go from listen state to timewait state? I thought a socket had to go through finwait1 and finwait2 to enter timewait state (which implicitly means only sockets who initiate graceful shutdown).

[ThadHouse]

Yes, on linux a listener socket itself can go into timewait. Its a bit weird. But it was extremely easy to reproduce.

[csujedihy]

Mind sharing how it could be repro'd?

Just create a listener socket and have a connection accepted and kill both the accepted connection and the listener socket itself? Then bind to the same listening port will fail?

[ThadHouse]

Yup. Have the listener socket accept a connection, and then kill the listener side. The listener side then will not be able to restart due to a bind failure.

[csujedihy]

I am really surprised linux has this weird behavior as it violates RFC793.

[ThadHouse]

Wouldn't the listener socket follow the left side of that tree down to TIME_WAIT?

[csujedihy]

The socket that's accepted from the listener socket would follow the left side but the listener socket itself, in my understanding, would not. When you call accept() on a listener socket, you get a new socket (in SynRcvd or ESTAB state) and the listener itself should stay in listening state.

[ThadHouse]

It was a bit confusing to me too. libuv actually sets SO_REUSEADDR for both Client initiated sockets and listener sockets unconditionally, but at least for my use case I didn't need it set on the client side, and thats a little more invasive too.