Merge configuration object from multiple files (instead of one single file)

doc/api/esmvalcore.config.rst

@@ -1,13 +1,15 @@
+.. _api_configuration:
+
 Configuration
 =============
 
 This section describes the :py:class:`~esmvalcore.config` module.
 
-Config
-******
+CFG
+***
 
-Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.config.Config` object.
-The global configuration can be imported from the :py:mod:`esmvalcore.config` module as :py:data:`~esmvalcore.config.CFG`:
+Configuration of ESMValCore/Tool is done via :py:data:`~esmvalcore.config.CFG`
+object:
 
 .. code-block:: python
 
@@ -16,7 +18,6 @@ The global configuration can be imported from the :py:mod:`esmvalcore.config` mo
     Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
             'compress_netcdf': False,
             'config_developer_file': None,
-            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
             'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
             'exit_on_warning': False,
             'log_level': 'info',
@@ -30,9 +31,10 @@ The global configuration can be imported from the :py:mod:`esmvalcore.config` mo
                          'default': '~/default_inputpath'},
             'save_intermediary_cubes': False)
 
-The parameters for the user configuration file are listed :ref:`here <user configuration file>`.
+All configuration parameters are listed :ref:`here <config_options>`.
 
-:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few extra functions, similar to :py:data:`matplotlib.rcParams`.
+:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few
+extra functions, similar to :py:data:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -41,8 +43,10 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
-All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
+Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path
+to an instance of :class:`pathlib.Path` and expands the home directory.
+All values entered into the config are validated to prevent mistakes, for
+example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
 
@@ -56,43 +60,53 @@ Or, if the value entered cannot be converted to the expected type:
     >>> CFG['max_parallel_tasks'] = '🐜'
     InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-:py:class:`~esmvalcore.config.Config` is also flexible, so it tries to correct the type of your input if possible:
+:py:data:`~esmvalcore.config.CFG` is also flexible, so it tries to correct the
+type of your input if possible:
 
 .. code-block:: python
 
     >>> CFG['max_parallel_tasks'] = '8'  # str
     >>> type(CFG['max_parallel_tasks'])
     int
 
-By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
-If it does not exist, it falls back to the default values.
-to load a different file:
+By default, the configuration is loaded from YAML files in the user's home
+directory at ``~/.config/esmvaltool``.
+If set, this can be overwritten with the ``ESMVALTOOL_CONFIG_DIR`` environment
+variable.
+Defaults for options that are not specified explicitly are listed :ref:`here
+<config_options>`.
+To reload the current configuration object according to these rules, use:
 
 .. code-block:: python
 
-    >>> CFG.load_from_file('~/my-config.yml')
+    >>> CFG.reload()
 
-Or to reload the current config:
+To load the configuration object from custom directories, use:
 
 .. code-block:: python
 
-    >>> CFG.reload()
+    >>> dirs = ['my/default/config', 'my/custom/config']
+    >>> CFG.load_from_dirs(dirs)
 
 
 Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the :py:data:`~esmvalcore.config.Session` object.
-A :py:data:`~esmvalcore.config.Session` can be initiated from the global :py:class:`~esmvalcore.config.Config`.
+This behavior can be controlled via the :py:data:`~esmvalcore.config.Session`
+object.
+A :py:data:`~esmvalcore.config.Session` must always be initiated from the
+global :py:data:`~esmvalcore.config.CFG` object:
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
 A :py:data:`~esmvalcore.config.Session` is very similar to the config.
-It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.config.Config`.
-At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.config.CFG`:
+It is also a dictionary, and copies all the keys from the
+:py:data:`~esmvalcore.config.CFG` object.
+At this moment, ``session`` is essentially a copy of
+:py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
@@ -102,7 +116,8 @@ At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.confi
     >>> print(session == CFG)  # False
     False
 
-A :py:data:`~esmvalcore.config.Session` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.config.Session` also knows about the directories where
+the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -118,7 +133,8 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.config.Config`.
+Unlike the global configuration, of which only one can exist, multiple sessions
+can be initiated from :py:data:`~esmvalcore.config.CFG`.
 
 
 API reference

doc/contributing.rst

@@ -101,7 +101,7 @@ Please keep the following considerations in mind when programming:
   code.
 - If you find yourself copy-pasting a piece of code and making minor changes
   to every copy, instead put the repeated bit of code in a function that you can
-  re-use, and provide the changed bits as function arguments.
+  reuse, and provide the changed bits as function arguments.
 - Be careful when changing existing unit tests to make your new feature work.
   You might be breaking existing features if you have to change existing tests.
 
@@ -577,7 +577,7 @@ users.
 
 When making changes, e.g. to the :ref:`recipe format <recipe_overview>`, the
 :ref:`diagnostic script interface <interfaces>`, the public
-:ref:`Python API <api>`, or the :ref:`configuration file format <config>`,
+:ref:`Python API <api>`, or the :ref:`configuration format <config>`,
 keep in mind that this may affect many users.
 To keep the tool user friendly, try to avoid making changes that are not
 backward compatible, i.e. changes that require users to change their existing
@@ -743,15 +743,15 @@ Perform the steps listed below with two persons, to reduce the risk of error.
    `PyPI <https://pypi.org/project/ESMValCore/>`__, and
    `readthedocs <https://readthedocs.org/dashboard/esmvalcore/users/>`__.
 
-The release of ESMValCore is tied to the release of ESMValTool. 
+The release of ESMValCore is tied to the release of ESMValTool.
 The detailed steps can be found in the ESMValTool
 :ref:`documentation <esmvaltool:release_steps>`.
-To start the procedure, ESMValCore gets released as a 
+To start the procedure, ESMValCore gets released as a
 release candidate to test the recipes in ESMValTool. If bugs are found
-during the testing phase of the release candidate, make as many release 
-candidates for ESMValCore as needed in order to fix them. 
+during the testing phase of the release candidate, make as many release
+candidates for ESMValCore as needed in order to fix them.
 
-To make a new release of the package, be it a release candidate or the final release, 
+To make a new release of the package, be it a release candidate or the final release,
 follow these steps:
 
 1. Check that all tests and builds work
@@ -795,13 +795,13 @@ Use the script
 to create create a draft of the release notes.
 This script uses the titles and labels of merged pull requests since the
 previous release.
-Open a discussion to allow members of the development team to nominate pull 
-requests as highlights. Add the most voted pull requests as highlights at the 
-beginning of changelog. After the highlights section, list any backward 
-incompatible changes that the release may include. The 
+Open a discussion to allow members of the development team to nominate pull
+requests as highlights. Add the most voted pull requests as highlights at the
+beginning of changelog. After the highlights section, list any backward
+incompatible changes that the release may include. The
 :ref:`backward compatibility policy<esmvaltool:backward-compatibility-policy>`.
-lists the information that should be provided by the developer of any backward 
-incompatible change. Make sure to also list any deprecations that the release 
+lists the information that should be provided by the developer of any backward
+incompatible change. Make sure to also list any deprecations that the release
 may include, as well as a brief description on how to upgrade a deprecated feature.
 Review the results, and if anything needs changing, change it on GitHub and
 re-run the script until the changelog looks acceptable.

doc/develop/fixing_data.rst

@@ -329,9 +329,9 @@ severity. From highest to lowest:
 
 Users can have control about which levels of issues are interpreted as errors,
 and therefore make the checker fail or warnings or debug messages.
-For this purpose there is an optional command line option `--check-level`
-that can take a number of values, listed below from the lowest level of
-strictness to the highest:
+For this purpose there is an optional :ref:`configuration option
+<config_options>` ``check_level`` that can take a number of values, listed
+below from the lowest level of strictness to the highest:
 
 - ``ignore``: all issues, regardless of severity, will be reported as
   warnings. Checker will never fail. Use this at your own risk.
@@ -375,8 +375,8 @@ To allow ESMValCore to locate the data files, use the following steps:
 
    - If you want to use the ``native6`` project (recommended for datasets whose
      input files can be easily moved to the usual ``native6`` directory
-     structure given by the ``rootpath`` in your :ref:`user configuration
-     file`; this is usually the case for native reanalysis/observational
+     structure given by the :ref:`configuration option <config_options>`
+     ``rootpath``; this is usually the case for native reanalysis/observational
      datasets):
 
      The entry ``native6`` of ``config-developer.yml`` should be complemented
@@ -399,17 +399,17 @@ To allow ESMValCore to locate the data files, use the following steps:
 
      To find your native data (e.g., called ``MYDATA``) that is for example
      located in ``{rootpath}/MYDATA/amip/run1/42-0/atm/run1_1979.nc``
-     (``{rootpath}`` is ESMValTool's ``rootpath`` for the project ``native6``
-     defined in your :ref:`user configuration file`), use the following dataset
+     (``{rootpath}`` is ESMValTool's ``rootpath`` :ref:`configuration option
+     <config_options>` for the project ``native6``), use the following dataset
      entry in your recipe
 
      .. code-block:: yaml
 
         datasets:
           - {project: native6, dataset: MYDATA, exp: amip, simulation: run1, version: 42-0, type: atm}
 
-     and make sure to use the following DRS for the project ``native6`` in your
-     :ref:`user configuration file`:
+     and make sure to use the following :ref:`configuration option
+     <config_options>` ``drs``:
 
      .. code-block:: yaml
 
@@ -437,9 +437,8 @@ To allow ESMValCore to locate the data files, use the following steps:
 
      To find your ICON data that is for example located in files like
      ``{rootpath}/amip/amip_atm_2d_ml_20000101T000000Z.nc`` (``{rootpath}`` is
-     ESMValTool ``rootpath`` for the project ``ICON`` defined in your
-     :ref:`user configuration file`), use the following dataset entry in your
-     recipe:
+     ESMValCore's :ref:`configuration option <config_options>` ``rootpath`` for
+     the project ``ICON``), use the following dataset entry in your recipe:
 
      .. code-block:: yaml