Separated control and compute loop, shorten the critical path, and enable more complicated policies

[Ying1123]

Moved from #1182. @xiezhq-hermann @hnyls2002
(I'm sorry that the original PR has been closed by me accidentally.)

Motivation

The existing design of the scheduler coupled control logic and model computation. While it simplifies the implementation, the overhead of scheduling can sometimes be non-negligible, especially when interacting with radix tree.
This PR decoupled the control and compute loops into different threads, overlapping computation and tasks that are not necessary on the critical paths. This reduces the scheduling overhead and enables more complicated policies in the future to be implemented on the control plane.
Based on the latest CI benchmark result, the PR would introduce about 10% throughput gain by reducing the overhead on critical path.

Modifications

1. Serialize access to shared resources including radix tree and memory pools with minimal overhead for safety.
2. A control loop that prepares the next prefill batch, handles finished requests and updates radix tree, concurrently with the compute loop.
3. Making policy scheduler compatible with the new design (to be further refined for corner cases).

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

What is the status of this PR? Do we still plan to merge it?

[xiezhq-hermann]

What is the status of this PR? Do we still plan to merge it?

I think so, @hnyls2002 did a fix on the case of tp_size > 1, which should behave the same as before and serve as a temporary fix. For the single GPU setting, it achieves 10%-15% throughput gain. We can always wait to have a more complete PR to merge, but I would suggest doing it incrementally before it diverges too much.
We might still need some style change before merging so please help review the PR. @merrymercy @Ying1123