Merge configuration object from multiple files (instead of one single file)

[schlunma]

Description

This PR is the first step towards our new configuration system (see #2371). It allows reading the configuration from multiple files (and directories), similar to Dask's configuration. The different files are merged using dask.config.collect().

The following directories are considered (descending priority):

1. The directory specified via the --config_dir command line argument
2. User configuration directory (by default ~/.config/esmvaltool, but this can be changed with
   the ESMVALTOOL_CONFIG_DIR environment variable)
3. Default configuration in repository (esmvalcore/config/configurations/defaults)

Related to #2371
Closes #805
Closes #1433

Link to documentation:

• https://esmvaltool--2448.org.readthedocs.build/projects/ESMValCore/en/2448/quickstart/configure.html
• https://esmvaltool--2448.org.readthedocs.build/projects/ESMValCore/en/2448/api/esmvalcore.config.html

Deprecations (since v2.12.0, will be removed in v2.14.0)

Usage of single configuration file ~/.esmvaltool/config-user.yml

As mentioned above, ESMValTool will now read configuration directories (instead of one single file). By default, the directory ~/.config/esmvaltool will be considered (can be changed with command line argument --config_dir; see below). To switch to the new format, run

mkdir -p ~/.config/esmvaltool && mv ~/.esmvaltool/config-user.yml ~/.config/esmvaltool

Configuration option config_file / command line argument --config_file:

As mentioned above, ESMValTool will now read configuration directories (instead of one single file). Instead of specifying one file, specify a configuration directory with --config_dir. To switch to the new format, run

mkdir -p ~/.config/esmvaltool && mv <path/to/your/old/config_file.yml> <your/new/path>

Config.load_from_file():

Please Config.load_from_dirs() instead, e.g.

from pathlib import Path
config_file = Path('path/to/my/config_file.yml')
CFG.load_from_dirs([config_file.parent])

Session.config_dir:

There is no replacement for this attribute.

Specify extra_facets_dir as tuple:

Please use a list instead. For example,

CFG['extra_facets_dir'] = ('path/1', 'path/2')  # deprecated, do not use anymore
CFG['extra_facets_dir'] = ['path/1', 'path/2']  # preferred syntax

Backwards-incompatible change

In some cases, simply copying the old configuration file (e.g., from ~/.esmvaltool/config-user-yml) to the new location might not be enough. In some cases, it might be necessary to adapt drs and rootpath settings. For example, on Levante, it might be necessary to change

drs:  # deprecated, do not use anymore
  CMIP3: DKRZ
  CMIP5: DKRZ
  CMIP6: DKRZ
  CORDEX: BADC

to

drs:  # preferred syntax
  CMIP3: DKRZ
  CMIP5: DKRZ
  CMIP6: DKRZ
  CORDEX: BADC
  obs4MIPS: default

Due to the new way settings are merged across configuration files, nested dictionaries are properly updated now.

Example:

The files

drs:  # default config
  CMIP5: ESGF
  obs4MIPS: ESGF

and

drs:  # user config
  CMIP5: DKRZ
  CMIP6: DKRZ

were previously merged to

drs:  # final config
  CMIP5: DKRZ
  CMIP6: DKRZ

In the new syntax, they are merged to

drs:  # final config
  CMIP5: DKRZ
  CMIP6: DKRZ
  obs4MIPS: ESGF

Thus, in the example above, it is necessary to explicitly specify obs4MIPS: default.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible see above
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.77%. Comparing base (546937f) to head (fe299dc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2448      +/-   ##
==========================================
+ Coverage   94.70%   94.77%   +0.06%     
==========================================
  Files         249      249              
  Lines       14114    14218     +104     
==========================================
+ Hits        13367    13475     +108     
+ Misses        747      743       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bouweandela]

Nice work here @schlunma!

[bouweandela]

Would it be possible to do this in a backward compatible way? The deprecations are fine, but it would be nice if existing configurations kept working (at least for a while). For example, if ESMVALTOOL_CONFIG_DIR is not set and ~/.config/esmvaltool does not exist, we could just continue to read the values from ~/.esmvaltool/config-user.yml or the command line --config-file argument, right? Then people would only need to switch when they actually need the new functionality.

[schlunma]

Would it be possible to do this in a backward compatible way? The deprecations are fine, but it would be nice if existing configurations kept working (at least for a while). For example, if ESMVALTOOL_CONFIG_DIR is not set and ~/.config/esmvaltool does not exist, we could just continue to read the values from ~/.esmvaltool/config-user.yml or the command line --config-file argument, right? Then people would only need to switch when they actually need the new functionality.

Yes, 100% agree. That's why I already implemented that in exactly this way. This will work for two more minor versions, and only then it will raise an error.

EDIT: My solution is even stricter: if --config_file is given or ~/.esmvaltool/config-user.yml exists, the tool will always read those, completely ignoring the new settings.

[schlunma]

@bouweandela I think I addressed all your comments, could you have another look? Thanks!

[bouweandela]

Thanks for the massive effort @schlunma! I have a few more suggestions for improving the documentation and then I think we can ask some people to test

[bouweandela]

What is the plan for the ESMValTool copy, remove?

[schlunma]

Yes, I think that's the easiest solution. We don't want users to simply copy-paste the file anymore but rather use esmvaltool config ..., so it doesn't need be located in the repo anymore.

[bouweandela]

@ESMValGroup/userengagementteam Could you please try this out and provide us with feedback on the user-friendliness?