What to do with incomplete graphs

[astrojuanlu]

Let's say you have this:

(Source: https://github.com/kedro-org/kedro-viz/tree/main/demo-project)

If you want to run the left branch, you might type this:

$ kedro run --from-nodes=ingestion.apply_types_to_reviews --to-nodes=ingestion.combine_step

but that will give you a nasty error:

ValueError: Pipeline input(s) {'ingestion.prm_agg_companies'} not found in the DataCatalog

If, on the other hand, you run the right branch, kedro run --from-nodes=ingestion.apply_types_to_companies --to-nodes=ingestion.combine_step it will fail with another error No such file or directory: '...data/02_intermediate/typed_shuttles.pq'.

The two errors are different but the root cause is the same: you are trying to execute a node that needs inputs that haven't been generated.

We (re-)discovered this behavior while reviewing the upcoming slicing feature of Kedro-Viz kedro-org/kedro-viz#2036

At some point we discussed the possibility of making kedro run do some backtracking. But there are lots of implications.

[astrojuanlu]

From @noklam:

Few reflections on this:

• We shouldn't try to make the CLI cover 100% API of what the Python code API can do. This is impossible and will probably make the CLI more confusing than useful.
  • That said, is the current CLI too limited that certain common things cannot be achieved? What's the use of each CLI argument. For example, if I received new data of companies but not shuttles, I will run kedro run --from-inputs companies to generate output
  • If i need to get a certain output <some_output>, I will do `kedro run --to-outputs some_output
    Back-tracking is useful, particular in the pipeline resume/interactive context. But in any case we don't want it to be enable automatically.
  • The fact that a node can be run without error doesn't mean that it is a wanted behavior.
  • It's also dangerous to back-track automatically.
    • This gives user expectation that Kedro figure out everything for you, because it runs without error, but there are conscious decision that user need to make:
      • Take the most stupid example, let say I want an output target, if target exist already, should Kedro not run anything at all?
      • Or should it run from the closest possible node. But why? Why do we assume the last node has the most updated data and simply re-run the last node is sufficient?
      • The "freshness" of data is always external to Kedro, unless we have this information we can never guess it right for our users.
      • Maybe kedro run --no-cache?
      • You want the same mechanism regardless of the type of datasets, otherwise it's confusing. For MemoryDataset it feels obvious because data is not saved, I surely want it to backtrack.
        • Maybe then we can use a 2-step approach? If something is missing, we generate the proper command <- This should be the "pipeline resume" case already, just that there are error being thrown even earlier than that and Kedro don't have that to generate the resume suggestion.

And the main discrepancy right now is that kedro run and kedro viz slice in opposite logic.

From node A to node C

In Kedro run, it's kedro run --from-nodes A --to-nodes C, that is

pipeline.from_nodes(A).to_nodes(C)

In kedro viz, it means Downstream of A until C and all upstream of C, that is

pipeline.from_nodes(A).to_nodes(C) | [pipeline.to](http://pipeline.to/)_nodes(C)  # The or operator is not supported in kedro run

[astrojuanlu]

Quick comment:

Your point about the cache is related to the "run only missing" thing that has been discussed time and time again, see #221, #2307.

[astrojuanlu]

I contend that, at the very least, the errors should be more clear.

[Huongg]

thank you both for your thought on this issue. Very interesting discussion going on here 😄

Just to clarify on your point @noklam

In Kedro run, it's kedro run --from-nodes A --to-nodes C, that is

pipeline.from_nodes(A).to_nodes(C)
In kedro viz, it means Downstream of A until C and all upstream of C, that is

`pipeline.from_nodes(A).to_nodes(C) | pipeline.to_nodes(C) # The or operator is not supported in kedro

Is this what you have in your mind for the visualisation part?

If so,

• viz can make this work as i just did a quick POC with FE changes only. Viz doesn't need to go upstream from 2, it can downstream from 1 and stop at 2.
• The reason why viz currently goes fully upstream is because I believe that the run command generated,
  kedro run --from-nodes=1 --to-nodes=2, doesn't work in kedro as it needs the other input leafs to run (1a and 1b). Is that correct or am I misunderstanding something?

This is what viz currently shows as it does go upstream

[noklam]

The reason why viz currently goes fully upstream is because I believe that the run command generated,
kedro run --from-nodes=1 --to-nodes=2, doesn't work in kedro as it needs the other input leafs to run (1a and 1b). Is that correct or am I misunderstanding something?

I think this need clarification. kedro run --from-nodes=1 --to-nodes=2 slice exactly what you shown in the first screenshot. The command is doing what is supposed to do, and both screenshot could be a desired result depending on what the user actually want.

For example:

1. If I know that I received new "companies" data and I want to re-run the pipeline, I know that there is no need to re-run the review and shuttle data, then kedro run --from-nodes 1 --to-nodes 2 will do the job.
2. If I don't know what should be re-run, then I may go to viz. Visually I see that those are the upstream nodes, so I will want to run the 2nd screenshot instead.

Both cases are valid, but we need to be clear which use cases are we trying to solve. (I have a feeling that case 2 is what we have in mind). If that's the case, we shouldn't be stuck on having kedro run --from-nodes xxx --to-nodes because this is not the right direction as we already know Kedro CLI doesn't have the same flexibility. kedro run --nodes is the only filter that can be used to support all cases.

For one specific target DAGs, there are many combination of filters can generate the same result. I think this is partly the reason why Kedro CLI was designed the way it is, sometimes it's easier to filter from the output, sometimes it's easier to filter from the starting nodes depending on the goal. You rarely use more than 1 pipeline filter (excluding the --pipeline).

The CLI itself doesn't have the flexibility to support arbitrary slicing (unless you are using --nodes directly). Maybe the direction here is, we use Kedro Viz to test Case 2 to see if it's something that users need. If that turns out to be the case, we may consider alternative syntax (instead of breaking the current one) like kedro run --select +combine_step (See dbt syntax)

[noklam]

Back to the situation that @astrojuanlu mentioned here.

$ kedro run --from-nodes=ingestion.apply_types_to_reviews --to-nodes=ingestion.combine_step

but that will give you a nasty error:

ValueError: Pipeline input(s) {'ingestion.prm_agg_companies'} not found in the DataCatalog

Something that can improve:

1. This is conflict with Kedro early validation, the validation throw error as soon as possible so at this point Kedro doesn't have the full information what is needed. What we want is kind of DryRunner + SoftfailRunner, where this would give us the full information of what dataset is missing (I try to do this a while ago but stopped because I found out the refactoring work of Runner breaks DryRunner in 0.19)

Assume that we have this information, we can do backtracking properly. Then we have different choice:

1. In Kedro CLI, it should still fails, but maybe delay the dataset error a little bit, and generate the correct resume command.
2. In Kedro Viz, it may use the DryRunner approach to quickly validate if the run is possible, if not use the resume command instead.