Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework local cache to address "not yet computed a rollout plan" issue. #48

Merged
merged 3 commits into from
Sep 25, 2024

Commits on Sep 12, 2024

  1. Configuration menu
    Copy the full SHA
    21c4c1c View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Rework local cache to address "not yet computed a rollout plan" issue.

    The rollout dashboard keeps a local cache of all tasks it has seen, because
    retrieving *all* task instances from Airflow is expensive.  We do this
    during the first loop, but in subsequent runs, we retrieve tasks that have
    updated or modified after the last check's timestamp.  To this, we also add
    tasks that have started after the last check.
    
    In addition to that, we now linearize the tasks in case that any of the
    retrieved task lists contains the same task, but may have been updated
    between requests.  Linearization involves picking, for each task instance
    the object with the latest date (be it execution, start or end date).
    
    In addition to that, if the rollout plan is somehow retrieved but marked
    empty when the schedule task is complete, we re-retrieve it again.  This
    prevents the odd error where the task has completed but the XCom associated
    with the task (containing the plan) is not yet saved to the database (or
    at least it looks that way, because we're racing to get the value right
    after the task finished, but the value is not yet inserted stably into
    the database).
    
    Finally, this PR parallelizes the multiple requests that take place when
    the task retrieval is performed.  This reduces incremental update time to
    roughly half of what it used to be.
    DFINITYManu committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    e043da2 View commit details
    Browse the repository at this point in the history
  2. Comment fix.

    DFINITYManu committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    4916ee5 View commit details
    Browse the repository at this point in the history