Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking progress towards completion. #340

Open
ryzhyk opened this issue Nov 8, 2020 · 1 comment
Open

Tracking progress towards completion. #340

ryzhyk opened this issue Nov 8, 2020 · 1 comment

Comments

@ryzhyk
Copy link
Contributor

ryzhyk commented Nov 8, 2020

Is there a way to estimate "progress towards completion" in the context of either timely or differential? Let's say we pushed some updates to DD, advanced the timestamp and are now calling step_or_park in a loop waiting for the probe to report that all updates have successfully propagated through the pipeline. This can take a long time, e.g., when populating the pipeline with initial data or when processing a small update that triggers large amount of recomputation. In these situations it may be nice to comfort the user with a little progress bar showing approximately how much longer they have to wait.

@frankmcsherry, do you happen to have some magic up your sleeve that might help here? :)

@Kixiron, @RDambrosio016.

@frankmcsherry
Copy link
Member

There is a lingering PR #321 that I need to clean up before merging, which would expose progress information in a logging channel. Depending on what makes it in, it could report either the outstanding capabilities in the system (messages counts on channels, and retained capabilities in operators) or the frontier of outstanding capabilities ("what's holding up the system").

I think this is the most obvious form of progress to report; there is also scheduling information about how many times and for how long various operators have been executed, but timely doesn't know too much about whether they are making progress when invoked.

Operator can also use their own custom logging if there is a clearer notion of progress that the operators can invoke (perhaps they have internal queues they are working to drain; timely wouldn't know about them, but the operators can still hook into the logging and report them).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants