Checking to see if kedro is running and at what stage it is at?

[dpavlic]

We want to implement a very light API on our pipelines--primary information being whether kedro is still running, and if so, at what stage it is at (but really the big need to know if it has stopped). Is there a way to get this information from outside the kedro process so we can do up a small flask API, and if not, I wonder if anyone has a good idea on how to implement this?

[datajoely]

This may be a good situation to use something like the ELK stack or Prometheus + Grafana since they can do things like dashboards easily.

[dpavlic]

All we really need for now is to know the pipeline currently running and / or if kedro is even running. But I can't think of a simple way to check that since kedro doesn't expose any of that information to the outside processes.

[antonymilne]

Probably the simplest approach would be to use the after_node_run/after_pipeline_run etc. hooks. Take a look at https://github.com/mzjp2/kedro-local-notify (in particular https://github.com/mzjp2/kedro-local-notify/blob/master/kedro_local_notify/hook.py) for the general idea, although obviously what goes in the hooks depends exactly on the system you want to use. If you're not interested in doing this as a plugin (can be shared between projects) but just as a one-off all you need to do is make a file hooks.py like this:

class NotificationHooks:
    @hook_impl
    def after_pipeline_run(self):
        # do something to show that kedro has finished running

and then enable NotificationHooks in HOOKS in settings.py.