Performance improvement: recipe_extremes_wind_3h.yml

[bouweandela]

This issue keeps track of the performance improvements implemented for recipe_extremes_wind_3h.yml
reported in #1915 (comment) as part of the ESiWACE3 service project.

Settings

To run the recipe, the following settings are used:

~/.esmvaltool/config-user.yml

max_parallel_tasks: 1

~/.esmvaltool/dask.yml

cluster:
  type: dask_jobqueue.SLURMCluster
  queue: compute
  account: bd0854
  cores: 128
  memory: 256GiB
  processes: 32
  interface: ib0
  local_directory: /scratch/b/b381141/dask-tmp
  n_workers: 32
  walltime: '8:00:00'

Profiling

A smaller version of the recipe was used for profiling runs. It uses only the wind_fut variable.

Conda environment used for profiling: environment.yml.

The profiles attached below were created with py-spy using the command

py-spy record \
--idle \
--rate 10 \
--subprocesses \
--format speedscope \
esmvaltool run recipe_extremes_wind_3h.yml

and can be viewed with https://www.speedscope.app:

1. Initial run profile_recipe_extremes_wind_3h.json (40 minutes)
2. Faster Faster trivial equality checks for coordinates and arrays SciTools/iris#5691 profile
   ...
3. main iris and ESMValCore branch on 2024-09-12 runs including Load esmvalcore.dataset.Dataset objects in parallel using Dask #2517 and Save all files in a task at the same time to avoid recomputing intermediate results #2522: single node (8 minutes), 2 nodes (5:20 minutes), 4 nodes (3:20 minutes)