Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separated control and compute loop, shorten the critical path, and enable more complicated policies #1287

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open

Separated control and compute loop, shorten the critical path, and enable more complicated policies #1287

wants to merge 39 commits into from
+316 −127

Commits on Aug 22, 2024

  1. decorator <synchronized> for safe concurrent access to radix treee

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d8b7262 View commit details
    Browse the repository at this point in the history

  2. decorator <synchronized> to serialize allocation and deallocation ope… 

    …rations to memory pools
    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c5732ba View commit details
    Browse the repository at this point in the history

  3. to protect the referenced nodes from eviction

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    59284a7 View commit details
    Browse the repository at this point in the history

  4. separated control and compute loop, shorten the critical path

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b11acb6 View commit details
    Browse the repository at this point in the history

  5. fix pre-commit

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ade5d86 View commit details
    Browse the repository at this point in the history

  6. bug fix due to interface and rebase error

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f105894 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into main

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a5fc4c8 View commit details
    Browse the repository at this point in the history

  8. fix tp issue

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    88f60c0 View commit details
    Browse the repository at this point in the history

  9. forcing empty commit

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ad2f4fe View commit details
    Browse the repository at this point in the history

  10. synchrnoize the gpu id to the compute thread for tp

    @xiezhq-hermann
    xiezhq-hermann committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a5e8bf5 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. style change; a more conservative pareto improvement instead of riski… 

    …ng performance for certain scenarios
    @xiezhq-hermann
    xiezhq-hermann committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f0298d5 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main'

    @xiezhq-hermann
    xiezhq-hermann committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8da3e00 View commit details
    Browse the repository at this point in the history

  3. fix precommit

    @xiezhq-hermann
    xiezhq-hermann committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1e5e62a View commit details
    Browse the repository at this point in the history

  4. style change

    @xiezhq-hermann
    xiezhq-hermann committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    602afe9 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into main

    @xiezhq-hermann
    xiezhq-hermann committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    fcd77eb View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. deepcopy for streaming only

    @xiezhq-hermann
    xiezhq-hermann committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    7d1864d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into main

    @hnyls2002
    hnyls2002 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    38d4b07 View commit details
    Browse the repository at this point in the history

  3. deepcopy for streaming only to reduce overhead

    @xiezhq-hermann
    xiezhq-hermann committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    ee521c3 View commit details
    Browse the repository at this point in the history

  4. minor

    @hnyls2002
    hnyls2002 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a7fb5c9 View commit details
    Browse the repository at this point in the history

  5. fix token alloc bug

    @hnyls2002
    hnyls2002 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    1b1a80c View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. fix sync when tp > 1

    @hnyls2002
    hnyls2002 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    e12de32 View commit details
    Browse the repository at this point in the history

  2. fix inconsistency when get_new_prefill_batch

    @hnyls2002
    hnyls2002 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    1c21664 View commit details
    Browse the repository at this point in the history

  3. fix phase_indicator

    @hnyls2002
    hnyls2002 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    2bf8fc9 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into xzq-pr

    @hnyls2002
    hnyls2002 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    48acd22 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. Merge branch 'main' into HEAD

    @Ying1123
    Ying1123 committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    9f93ca0 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review

    @merrymercy
    merrymercy committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    8e73479 View commit details
    Browse the repository at this point in the history

  3. Update python/sglang/srt/managers/io_struct.py

    @merrymercy
    merrymercy committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    67caec2 View commit details
    Browse the repository at this point in the history

  4. trigger CI

    @merrymercy
    merrymercy committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    74eaece View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into xiezhq-hermann/main

    @merrymercy
    merrymercy committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    8db88d1 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into xiezhq-hermann/main

    @xiezhq-hermann
    xiezhq-hermann committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    c05cfd1 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into xiezhq-hermann/main

    @merrymercy
    merrymercy committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    47f0f49 View commit details
    Browse the repository at this point in the history

  8. single thread when tp

    @hnyls2002
    hnyls2002 committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    aed7426 View commit details
    Browse the repository at this point in the history

  9. reduce queue.Queue overhead

    @hnyls2002
    hnyls2002 committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    a1a2c08 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. optimize streaming when tp

    @hnyls2002
    hnyls2002 committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    ab94e71 View commit details
    Browse the repository at this point in the history

  2. fix

    @hnyls2002
    hnyls2002 committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    6077ed5 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. fix typo

    @hnyls2002
    hnyls2002 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f4be82c View commit details
    Browse the repository at this point in the history

  2. fix

    @hnyls2002
    hnyls2002 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    c38e200 View commit details
    Browse the repository at this point in the history

  3. reduce sleep time

    @hnyls2002
    hnyls2002 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    b8c46d4 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Merge branch 'main' into xiezhq-hermann/main

    @xiezhq-hermann
    xiezhq-hermann committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    1e853e5 View commit details
    Browse the repository at this point in the history