What does best practice in Kedro look like?

[datajoely]

A good question by was asked on Discord server about what best practice Kedro look like. This is something that will evolve and cannot and should not be designed up front.

However, we are now at a point where we there are thousands of Kedro practitioners and power users our there in the wild. At the time of writing this post - Kedro has been open sourced for ~2 years, we average about ~8,000 users visiting the Kedro documentation per month with only 10% of that traffic coming from internal users.

So - let's kick off a discussion where we share our own ideas of using best practice when using Kedro! 🚀

[datajoely]

Firstly - I would point everyone towards the Principles of Kedro in the docs which articulates the team's wider philosophy when building the framework and this applies to our thinking on how users should approach building out pipelines too.

From a personal perspective, I've been using what eventually became the Kedro open source project today for about 4 years and have a few personal views about what best practice could look like:

• Split your catalog files into multiple YAML files for maintainability
• Use TemplatedConfigLoader straight away
• Design code to be readable 6 months later - write code for someone else to read, even if that person is future you!
• Keep nodes simple and to the point
• Avoid dynamic DAG creation in Kedro unless you really have to
• If you have to, ensure that the DAG is structurally immutable and only differs in terms of dataset flow.
• Most extensions to Kedro can and should happen via a Hook
• Use pre-commit hooks in your workflow (check out what we do in the Kedro repo if you need inspiration)
• Write tests with fabricated data, use something like Great Expectations for defensive checks at run time

[pascalwhoop]

To expand on top of what you have listed, for running pipelines in production:

• Set up CICD on day 1
• start with a dummy pipeline running in production first
• Write a subpipeline that creates sample/test data from the production environment in your dev environment that gets run via CICD. No prod data leaves prod & test-runs are run on 1/n-th the data size
• great you have test data to play with now! -> unit tests!
• use docker
• run your pipeline with an orchestrator. kubeflow, airflow, argo, choose one and then
• size your nodes appropriately. Each node should be a significant step of processing, not just appending a column

I sketched a git-ops flow that worked for me in the past

[JacobJeppesen]

Well, I'm a newcomer trying out Kedro in these days, so I don't have any best practices to share. However, I do have a question regarding best practices. Are people just using raw strings when defining inputs/outputs of nodes in a large pipeline? I quickly end up with quite a number of these, which easily becomes a mess. And does anyone know of any good open source projects based on Kedro which can showcase some of the best practices? It would be great to be able to go through and see how they organize large pipelines, enforce naming schemas, runs pipeline testing, etc.

[datajoely]

Unfortunately all of our implementations of Kedro internally are not publically available. Designing a proper end to end 'project' has been on my backlog for a while and it just keep gets getting pushed...

In terms of simplifying Catalogs I have a few bits of advice:

• YAML anchors can reduce the amount of boilerplate
• Jinja2 can give you some programming constructs to essentially generate lots of config at runtime (but I personally prefer more readable YAML)

We also have a prototype mechanism for simplifying Catalog definitions as well. Would you be interested in participating in a UX research interview?

[arnaldog12]

One of the examples of best practices I liked the most was described in the kedro-mlflow plugin. Basically, it uses tags to reuse nodes. For example, when defining an ML pipeline, you can use the tags training and serving to define which node will be used in each step (or both steps).