Questions around this proposal

[benlesh]

I work on RxJS, and we have our own implemented schedulers (which frankly I hate, but I see the utility in them). I think it would be ideal to move to using this proposed API when it is more broadly available, however, I'm uncertain if it would meet some of our users' requirements (for better or worse).

1. Would setting the priority of a task to immediate place it in front of all existing non-immediate tasks? Including promise resolution etc?
2. How does this relate to and interact with requestAnimationFrame? Can this API be leveraged, at least for scheduling, in place of requestAnimationFrame?

[shaseley]

Hi Ben, thanks for reaching out!

Unfortunately the API in its current state won't meet your requirements. As a starting point, we've narrowed the scope of the scheduler to tasks that run in between frames, working towards a yieldy asynchronous task model. We do realize that this isn't the full picture, and want the scheduler to be something that evolves over time to cover more use cases.

1. Would setting the priority of a task to immediate place it in front of all existing non-immediate tasks? Including promise resolution etc?

immediate priority was removed a while back (current priorities are here). The closest is user-blocking, but all postTask tasks start in separate browser tasks, so microtasks have higher priority.

We have tossed around the idea of a microtask priority, similar to queueMicrotask but that uses signals, etc. This sounds like it wouldn't work for your use case either? I think this is related running multiple userspace tasks per browser task, which isn't the task model we were focusing on, but would be interested to hear more and how you think about these tasks.

2. How does this relate to and interact with requestAnimationFrame? Can this API be leveraged, at least for scheduling, in place of requestAnimationFrame?

No, this API currently only deals with inter-frame tasks and rAF runs tasks in frames. I do want to explore this relationship more in the future, and there are a couple aspects to this:

1. How should browsers prioritize between rendering tasks and postTask tasks (current priorities) (issue)? This is currently left up to browsers and not specified. The rendering steps (including rAF) can run between every postTask task (or non-postTask task) but isn't required to.
2. Incorporating frame-aligned tasks. As mentioned, this was out-of-scope for V1. This could be something we tackle if there is sufficient interest, but it's not clear what the scope/requirements should be, e.g. should we do read/write alignment? What's missing from requestAnimationFrame? Are there use cases for moving tasks between frame-aligned and not?

I think what I'd like to do is file individual issues for these and mark them as enhancements once I hear a bit more about your use cases.

[benlesh]

Our use case is more general. We need a unified API for scheduling tasks to happen at scheduler-implementation-defined points in time. Ideally, all of the "common cases" (microtasks, same-tick, animation-frame, etc) would be covered natively, and we wouldn't have to ship it. However, we have use cases for people to create and define their own scheduler that fits the same API, so they can control when tasks are fired.

Right now, RxJS has a few schedulers: QueueScheduler (basic FIFO queue task scheduling that starts flushing on first schedule), ASAPScheduler, (microtasks), AsyncScheduler, (setTimeout(0) scheduling), AnimationFrameScheduler, VirtualScheduler (a scheduler that needs to be flushed manually), TestScheduler, a specialized VirtualScheduler that is used to create test scenarios to test async actions deterministically.

All of these schedulers allow scheduling a 0 to get the default behavior, or scheduling a a later time (>0), which generally falls back to a setTimeout behavior. This aspect of the design, frankly, is rife with subtle flaws. In practice, users rarely if ever hit these problems, but it would be nice if a better API existed natively, and we could get people moving that direction. What I sort of wish is that we could have a native scheduler that attempts to realize that there is a relationship between all asynchronously scheduled things, whether we're talking about near-term event loop (microtasks, etc), or long-term event loops (setTimeout, setInterval, requestAnimationFrame, etc).... and set up a future where the API around these things could all be the same shape.

[Jamesernator]

2. Incorporating frame-aligned tasks. As mentioned, this was out-of-scope for V1. This could be something we tackle if there is sufficient interest, but it's not clear what the scope/requirements should be, e.g. should we do read/write alignment? What's missing from requestAnimationFrame? Are there use cases for moving tasks between frame-aligned and not?

I think the main thing missing from requestAnimationFrame is tight integration with async/await patterns that scheduler.* patterns enable. In particular it would nice to be able to do:

async function renderLoop() {
  while (true) {
    const updatesToApply = await prepareForFrame();
    // This could be done with new Promise(requestAnimationFrame)
    // but it would make sense to have similar yielding to what would
    // be happening inside prepareFrameContent
    await scheduler.requestAnimationFrame(() => {
        // Update state and stuff synchronously
    });
  }
}

What would be particularly helpful too is an analogue of IdleDeadline but for frames, essentially a deadline as to when the next frame must be ready. This would make concurrency logic for things like prepareForFrame() a lot simpler, as they could detect how urgent particular work is.

Such a thing may even make more sense as it's own kind've scheduling, for example something like:

async function prepareForFrame(frameScheduler) {
  const updates = [];
  for (const entity of gameEntities) {
    updates.push(...await computeNewEntityLocation(frameScheduler));
    // If there's user blocking events, i.e. scroll, between frames
    // we will yield to them, however the browser would still schedule
    // this with rather high priority as it'll be needed for the next frame
    await frameScheduler.yield();
  }
  // etc etc
}

async function gameLoop() {
  while (true) {
    // Resolves when the entire frame is complete
    await scheduler.requestFrameSchedule(async (frameScheduler) => {
      // This schedules for same time as requestAnimationFrame does today
      // others might be interested in scheduleAfterFrame as there does seem
      // to be interest in requestPostAnimationFrame
      frameScheduler.scheduleImmediatelyBeforeFrame((frameHiResTime) => {
        applyUpdates(updatesToApply, frameHiResTime);
      });
      const updatesToApply = await prepareForFrame(frameScheduler);
    });
  }
}

There might even be extensions to this that give the browsers considerably more hints about priority, e.g. suppose we're looping over something we could send the scheduler yield hints to say there's such and such items remaining, if the browser saw that consecutive loops were fast and there's few loops remaining it might put it on lower priority i.e.

async function updateEntitites(frameScheduler) {
  for (const [index, entity] of gameEntities.entries()) {
    // ... compute stuff here ...

    // The browser would sample the time between waking this and the next yield
    // to decide if it needs to prioritize this loop or should switch to other loops
    // before the deadline is up
    // The browser would essentially keep a running average of times between wakeup
    // and the next yield to determine if these tasks are high priority or not
    // i.e. If we have 10ms remaining in our frame deadline and there's two tasks
    // remaining but the previous tasks took 3ns or something, then they would be
    // put on low priority
    await frameScheduler.yield({ yieldsRemaining: gameEntities.length - index });
  }
}

This could of course be taken even further, but there's obviously the fact that more complex scheduling power would take more to compute. The above suggestion would work fairly well for tasks of similar size OR in cases where the tasks are sorted roughly longest to shortest (as longest->shortest means average time between tasks decreases, so early worst case estimates become better closer to the frame and the scheduler can progressively deprioritize these tasks to do other work).

[nhelfman]

@Jamesernator, I find these ideas very interesting and creative. However, I feel that giving such granular control will increase confusion about these concepts among developers (scheduling is already a non-trivial topic).

Do you feel that these use cases could be achieved as well, perhaps with a bit more verbose code, by using the simpler proposal isFramePending() API?

[Jamesernator]

Do you feel that these use cases could be achieved as well, perhaps with a bit more verbose code, by using the simpler proposal isFramePending() API?

Not really, because you'd also have to have .isInputPending() in the loop to check if you need to yield, but if you do this you can't guarantee you'll be woken before the next frame i.e.:

async function updateEntities() {
  for (const entity of entities) {
    updateEntitySomehow(entity);

    // If a frame is pending, we DO NOT yield so that we can complete our work
    // But if input is pending we want to allow the events to fire and so on
    if (scheduler.isInputPending() && !scheduler.isFramePending()) {
      // But what should we even yield to here? Probably user-blocking, but 
      // does this guarantee being woken before the next frame? Probably not guaranteed
      // in general that this will resume before next frame, and if this isn't guaranteed, 
      // then we can't detect stale frames in any reliably obvious way
      await scheduler.yield("user-blocking");
    }
  }
}

async function gameLoop() {
  while (true) {
    await updateEntities();
    drawFrame();
  }
}

A simpler API might be possible though, simply a method that schedules a yield that is guaranteed to execute before the frame is run.

async function updateEntities() {
  for (const entity of entities) {
    // If the frame is actually pending then we don't yield, we complete our work and
    // don't yield anymore
    if (scheduler.isInputPending() && !scheduler.isFramePending()) {
      // Yield to the frame, right before the frame all pending .yield("next-frame") MUST
      // be woken so that any work needed to complete can be finished, this can still
      // be woken at any earlier point though during idle periods or when
      // frame work is highest priority
      await scheduler.yield("before-next-frame");
    }
  }
}

async function gameLoop() {
  while (true) {
    // This is THE SAME as before
    // If a frame is pending, we DO NOT yield so that we can complete our work
    if (scheduler.isInputPending() && !scheduler.isFramePending()) {
      await updateEntities();
      drawFrame();
    }
  }
}

[Jamesernator]

This could actually be generalized to all the main yield points one probably cares about in the rendering cycle:

// Schedules for ANYTIME BEFORE the next frame, if idle
// these will be executed as soon as possible
await scheduler.yield("anytime-before-next-frame");
// Schedules for IMMEDIATELY BEFORE the next frame, this
// is essentially what requestAnimationFrame does today
await scheduler.yield("pre-frame");
// Schedules for IMMEDIATELY AFTER the next frame, this
// is essentially what requestPostAnimationFrame does today
await scheduler.yield("post-frame");

[mmocny]

(The @Jamesernator thread seems much removed from the original @benlesh topic, and I suggest forking this issue at this point since the RxJS topic seems exhausted.)

---

Some of the ideas @Jamesernator presented are super intriguing.

Specifically, having a way to hint that a task is specifically meant to be scheduled before the next animation frame, but without just blocking the next animation frame, is a feature that doesn't exist right now.

• "pre-frame" is already available via requestAnimationFrame api.
  • We might want a promise version of this api (requested often!), and
  • We might want to split the two jobs of "request schedule next frame" and "fire event on frame start"
    • And even there you might want to support {once:} with false (i.e. create an Observable stream, not just a single Promise/callback)
  • But generally you can somewhat polyfill these with rAF.
• "post-frame" you also have.
  • Either via rAF + postTask, or
  • via the yield({ afterRender: true }) proposal.

Today, the way to do "anytime-before-next-frame" is probably to just block (i.e. from inside an event handler), or to block until some deadline (i.e. from inside a requestAnimationFrame callback), and likely without yield points.

The current alternative is to just schedule tasks (perhaps with user-blocking), and hope that as many are scheduled before next animation frame as possible, and abort() the rest.

But, there are some scheduling experiments that @shaseley is working on where we might explicitly try to delay tasks whenever next animation frame is requested (especially after interactions). Knowing which tasks are meant before the animation frame, yet still supporting yield, could be useful. (But, maybe thats what user-blocking priority is?)