Using multiple parameter objects as inputs to a single node

[hika-ru]

Hi, I'm trying to simultaneously make use of pipeline specific configurations in conf/base/parameters/pipeline_name.yml, as well as configurations that are relevant to all pipelines and set in conf/base/parameters.yml.

But how can I specify multiple different parameters inputs for the node objects inside a pipeline.py module? The following didn't work for me:

node(
  ...,
  inputs=[
    "model_input_table",
    "params:param_dict", 
    "more_params:global_param_dict",
  ],
  ...,
)

Is params a keyword that can only be used once for the inputs of a single node? Is it at all possible to decouple pipeline specific and global configuration parameters in the way that I'm trying? Thank you for your help!

[hika-ru]

Found the answer myself after reading the documentation on the inputs argument and thinking very hard! inputs can be a dict, where the keys correspond to the names of the node function's argument names. The example from the original question can be solved as follows:

node(
  ...,
  func=do_something,
  inputs={
    "data": "model_input_table",
    "parameters": "params:param_dict", 
    "more_params": "params:global_param_dict",
  },
  ...,
)

The given inputs will be applied nicely to the arguments in:

def do_something(data: pd.DataFrame, parameters: dict, more_params: dict) -> Any:
  ...

[antonymilne]

Just to point out there's no particular need to use a dictionary for inputs as keyword arguments here. It will still work just as a list that gives positional arguments in the right order:

node(
  ...,
  func=do_something,
  inputs=[
    "model_input_table",
    "params:param_dict", 
    "params:global_param_dict",
  ],
  ...,
)

You can use parameters or params:... as many times you like as a node input, just like a catalog entry name. This will be mapped in the order they are given to function arguments (or you can do it explicitly in any order if you use kwargs in a dictionary).

[hika-ru]

Ohh, I see... I read to much into the inner workings after having done the spaceflight tutorial, but actually it's much simpler 😃 . Thank you!