Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to convert units issue with recipe_climwip_brunner2019_med.yml #3694

Open
ehogan opened this issue Jun 27, 2024 · 5 comments
Open

Unable to convert units issue with recipe_climwip_brunner2019_med.yml #3694

ehogan opened this issue Jun 27, 2024 · 5 comments
Labels
bug

Comments

@ehogan
Copy link
Contributor

ehogan commented Jun 27, 2024 •
edited
Loading

Describe the bug

Traceback (most recent call last):
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_main.py", line 533, in run
    fire.Fire(ESMValTool())
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_main.py", line 413, in run
    self._run(recipe, session)
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_main.py", line 455, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_main.py", line 130, in process_recipe
    recipe.run()
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_recipe/recipe.py", line 1095, in run
    self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_task.py", line 738, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_task.py", line 782, in _run_parallel
    _copy_results(task, running[task])
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_task.py", line 805, in _copy_results
    task.output_files, task.products = future.get()
                                       ^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/miniforge3/envs/new-env/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 684, in _run
    product.apply(step, self.debug)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 492, in apply
    self.cubes = preprocess(self.cubes, step,
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in cubes
    self._cubes = [ds.load() for ds in self.datasets]
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in <listcomp>
    self._cubes = [ds.load() for ds in self.datasets]
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/dataset.py", line 685, in load
    cube = self._load()
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/dataset.py", line 771, in _load
    result = preprocess(
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 397, in preprocess
    result.append(_run_preproc_function(function, items, settings,
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 346, in _run_preproc_function
    return function(items, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/cmor/fix.py", line 195, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 425, in fix_metadata
    self._fix_units(cube)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 413, in _fix_units
    cube.convert_units(self.vardef.units)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/iris/cube.py", line 1460, in convert_units
    new_data = _lazy.lazy_elementwise(self.lazy_data(), pointwise_convert)
    ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/iris/_lazy_data.py", line 448, in lazy_elementwise
    dtype = elementwise_op(np.zeros(1, lazy_array.dtype)).dtype
      ^^^^^^^^^^^^^^^^^
  File "/miniforge3/envs/new-env/lib/python3.11/site-packages/cf_units/__init__.py", line 1918, in convert
    raise ValueError(
ValueError: Unable to convert from 'Unit('1.15740740740741e-05 kg.s-4')' to 'Unit('W m-2')'.

@ESMValGroup/technical-lead-development-team would any of you have any indication of whether one of the recently added PRs may have caused this error? 🤔
@valeriupredoi
Copy link
Contributor

could be cf_units - latest release is from about a month ago https://anaconda.org/conda-forge/cf-units

This was referenced Jul 1, 2024
Closed
Closed
@bouweandela bouweandela mentioned this issue Jul 2, 2024
Closed
@bouweandela
Copy link
Member

@ehogan Would you be able to share a bit more of the stack trace?

@ehogan
Copy link
Contributor Author

ehogan commented Jul 2, 2024

@ehogan Would you be able to share a bit more of the stack trace?

Apologies for the delay; I have updated the first comment of this issue with the full traceback 👍

@bouweandela
Copy link
Member

bouweandela commented Jul 2, 2024 •
edited
Loading

It looks like this happens for variable rsus. On Levante, the rsus data is e.g. /work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1995_monthly.nc. In the ERA5 documentation, Mean surface net short-wave radiation flux is listed in Table 4 and not in Table 3 which lists accumulated variables. Yet, in the fix it is assumed that this is an accumulation: https://github.com/ESMValGroup/ESMValCore/blob/60aec3e5abe9a58db2f7256e979e2f4a888a0296/esmvalcore/cmor/_fixes/native6/era5.py#L285.

Therefore I suspect that either this is the wrong input data (maybe rsus maps to another variable?) or the fix is wrong (the call to fix_accumulated_units should be removed).

Complete error message: 
ERROR   [829235] Failed to run preprocessor function 'fix_metadata' on the data
[<iris 'Cube' of surface_upwelling_shortwave_flux_in_air / (1.15740740740741e-05 kg.s-4) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>]
(and 16 further argument(s) not shown here; refer to the debug log for a full list)
loaded from original input file(s)
[LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1995_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1996_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1997_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1998_monthly.nc')]
(and 16 further file(s) not shown here; refer to the debug log for a full list)
with function argument(s)
check_level = <CheckLevels.DEFAULT: 3>,
session = Session({'auxiliary_data_dir': PosixPath('/work/bd0854/DATA/ESMValTool2/AUX'),
         'check_level': <CheckLevels.DEFAULT: 3>,
         'compress_netcdf': False,
         'config_developer_file': PosixPath('/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/config-developer.yml'),
         'config_file': PosixPath('/home/b/b382148/.esmvaltool/config-user.yml'),
         'diagnostics': None,
         'download_dir': PosixPath('/work/bd0854/DATA/ESMValTool2/download'),
         'drs': {'CMIP3': 'DKRZ',
                 'CMIP5': 'DKRZ',
                 'CMIP6': 'DKRZ',
                 'CORDEX': 'BADC',
                 'OBS': 'default',
                 'OBS6': 'default',
                 'ana4mips': 'default',
                 'native6': 'default',
                 'obs4MIPs': 'default'},
         'exit_on_warning': False,
         'extra_facets_dir': (),
         'log_level': 'info',
         'max_datasets': None,
         'max_parallel_tasks': 8,
         'max_years': None,
         'output_dir': PosixPath('/scratch/b/b382148/testing_v2.11.0rc2/run1/esmvaltool_output'),
         'output_file_type': 'png',
         'profile_diagnostic': False,
         'remove_preproc_dir': True,
         'resume_from': [],
         'rootpath': {'CMIP3': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP3')],
                      'CMIP5': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ')],
                      'CMIP6': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ')],
                      'CORDEX': [PosixPath('/work/ik1017/C3SCORDEX/data/c3s-cordex/output')],
                      'OBS': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'OBS6': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'RAWOBS': [PosixPath('/work/bd0854/DATA/ESMValTool2/RAWOBS')],
                      'ana4mips': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'native6': [PosixPath('/work/bd0854/DATA/ESMValTool2/RAWOBS')],
                      'obs4MIPs': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')]},
         'run_diagnostic': True,
         'save_intermediary_cubes': False,
         'search_esgf': 'when_missing',
         'skip_nonexistent': False,
         'write_ncl_interface': False}),
short_name = 'rsus',
mip = 'Amon',
preprocessor = 'region_std',
derive = True,
force_derivation = False,
dataset = 'ERA5',
project = 'native6',
type = 'reanaly',
version = 'v1',
tier = 3,
timerange = '1995/2014',
variable_group = 'rsus_STD',
diagnostic = 'calculate_weights_climwip',
recipe_dataset_index = 77,
alias = 'native6',
original_short_name = 'rsus',
standard_name = 'surface_upwelling_shortwave_flux_in_air',
long_name = 'Surface Upwelling Shortwave Radiation',
units = 'W m-2',
modeling_realm = ['atmos'],
frequency = 'mon'
ERROR   [829234] Failed to run preprocessor function 'fix_metadata' on the data
[<iris 'Cube' of surface_upwelling_shortwave_flux_in_air / (1.15740740740741e-05 kg.s-4) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>,
 <iris 'Cube' of Mean surface net short-wave radiation flux / (W m**-2) (time: 12; latitude: 721; longitude: 1440)>]
(and 16 further argument(s) not shown here; refer to the debug log for a full list)
loaded from original input file(s)
[LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1995_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1996_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1997_monthly.nc'),
 LocalFile('/work/bd0854/DATA/ESMValTool2/RAWOBS/Tier3/ERA5/v1/mon/rsus/era5_mean_surface_net_short_wave_radiation_flux_1998_monthly.nc')]
(and 16 further file(s) not shown here; refer to the debug log for a full list)
with function argument(s)
check_level = <CheckLevels.DEFAULT: 3>,
session = Session({'auxiliary_data_dir': PosixPath('/work/bd0854/DATA/ESMValTool2/AUX'),
         'check_level': <CheckLevels.DEFAULT: 3>,
         'compress_netcdf': False,
         'config_developer_file': PosixPath('/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/config-developer.yml'),
         'config_file': PosixPath('/home/b/b382148/.esmvaltool/config-user.yml'),
         'diagnostics': None,
         'download_dir': PosixPath('/work/bd0854/DATA/ESMValTool2/download'),
         'drs': {'CMIP3': 'DKRZ',
                 'CMIP5': 'DKRZ',
                 'CMIP6': 'DKRZ',
                 'CORDEX': 'BADC',
                 'OBS': 'default',
                 'OBS6': 'default',
                 'ana4mips': 'default',
                 'native6': 'default',
                 'obs4MIPs': 'default'},
         'exit_on_warning': False,
         'extra_facets_dir': (),
         'log_level': 'info',
         'max_datasets': None,
         'max_parallel_tasks': 8,
         'max_years': None,
         'output_dir': PosixPath('/scratch/b/b382148/testing_v2.11.0rc2/run1/esmvaltool_output'),
         'output_file_type': 'png',
         'profile_diagnostic': False,
         'remove_preproc_dir': True,
         'resume_from': [],
         'rootpath': {'CMIP3': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP3')],
                      'CMIP5': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ')],
                      'CMIP6': [PosixPath('/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ')],
                      'CORDEX': [PosixPath('/work/ik1017/C3SCORDEX/data/c3s-cordex/output')],
                      'OBS': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'OBS6': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'RAWOBS': [PosixPath('/work/bd0854/DATA/ESMValTool2/RAWOBS')],
                      'ana4mips': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')],
                      'native6': [PosixPath('/work/bd0854/DATA/ESMValTool2/RAWOBS')],
                      'obs4MIPs': [PosixPath('/work/bd0854/DATA/ESMValTool2/OBS')]},
         'run_diagnostic': True,
         'save_intermediary_cubes': False,
         'search_esgf': 'when_missing',
         'skip_nonexistent': False,
         'write_ncl_interface': False}),
short_name = 'rsus',
mip = 'Amon',
preprocessor = 'region_mean',
derive = True,
force_derivation = False,
dataset = 'ERA5',
project = 'native6',
type = 'reanaly',
version = 'v1',
tier = 3,
timerange = '1995/2014',
variable_group = 'rsus_CLIM',
diagnostic = 'calculate_weights_climwip',
recipe_dataset_index = 77,
alias = 'native6',
original_short_name = 'rsus',
standard_name = 'surface_upwelling_shortwave_flux_in_air',
long_name = 'Surface Upwelling Shortwave Radiation',
units = 'W m-2',
modeling_realm = ['atmos'],
frequency = 'mon'
INFO    [829044] Maximum memory used (estimate): 9.1 GB
INFO    [829044] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
ERROR   [829044] Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
                   ^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 684, in _run
    product.apply(step, self.debug)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 492, in apply
    self.cubes = preprocess(self.cubes, step,
                            ^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in cubes
    self._cubes = [ds.load() for ds in self.datasets]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in <listcomp>
    self._cubes = [ds.load() for ds in self.datasets]
                   ^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/dataset.py", line 685, in load
    cube = self._load()
           ^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/dataset.py", line 771, in _load
    result = preprocess(
             ^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 397, in preprocess
    result.append(_run_preproc_function(function, items, settings,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 346, in _run_preproc_function
    return function(items, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/fix.py", line 195, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 425, in fix_metadata
    self._fix_units(cube)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 413, in _fix_units
    cube.convert_units(self.vardef.units)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/iris/cube.py", line 1460, in convert_units
    new_data = _lazy.lazy_elementwise(self.lazy_data(), pointwise_convert)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/iris/_lazy_data.py", line 448, in lazy_elementwise
    dtype = elementwise_op(np.zeros(1, lazy_array.dtype)).dtype
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/cf_units/__init__.py", line 1918, in convert
    raise ValueError(
ValueError: Unable to convert from 'Unit('1.15740740740741e-05 kg.s-4')' to 'Unit('W m-2')'.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_main.py", line 533, in run
    fire.Fire(ESMValTool())
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_main.py", line 413, in run
    self._run(recipe, session)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_main.py", line 455, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_main.py", line 130, in process_recipe
    recipe.run()
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_recipe/recipe.py", line 1095, in run
    self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 738, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 782, in _run_parallel
    _copy_results(task, running[task])
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 805, in _copy_results
    task.output_files, task.products = future.get()
                                       ^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 684, in _run
    product.apply(step, self.debug)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 492, in apply
    self.cubes = preprocess(self.cubes, step,
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in cubes
    self._cubes = [ds.load() for ds in self.datasets]
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 502, in <listcomp>
    self._cubes = [ds.load() for ds in self.datasets]
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/dataset.py", line 685, in load
    cube = self._load()
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/dataset.py", line 771, in _load
    result = preprocess(
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 397, in preprocess
    result.append(_run_preproc_function(function, items, settings,
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 346, in _run_preproc_function
    return function(items, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/fix.py", line 195, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 425, in fix_metadata
    self._fix_units(cube)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 413, in _fix_units
    cube.convert_units(self.vardef.units)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/iris/cube.py", line 1460, in convert_units
    new_data = _lazy.lazy_elementwise(self.lazy_data(), pointwise_convert)
    ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/iris/_lazy_data.py", line 448, in lazy_elementwise
    dtype = elementwise_op(np.zeros(1, lazy_array.dtype)).dtype
      ^^^^^^^^^^^^^^^^^
  File "/home/b/b382148/miniforge3/envs/esmvaltool-core2.11.0rc2/lib/python3.11/site-packages/cf_units/__init__.py", line 1918, in convert
    raise ValueError(
ValueError: Unable to convert from 'Unit('1.15740740740741e-05 kg.s-4')' to 'Unit('W m-2')'.
INFO    [829044] 
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

@schlunma schlunma mentioned this issue Jul 3, 2024
Open
@schlunma
Copy link
Contributor

schlunma commented Jul 3, 2024

While looking at this, we found some very severe issues with the ERA5 CMOrizer: ESMValGroup/ESMValCore#2475

Summary: rsus and rlus are incorrect, and for other variables the derivation can be simplified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ehogan @bouweandela @valeriupredoi @schlunma