s5p-compress "some chunks keys are not dimensions on this object"

[LillaBW2327]

Hello, apologies for raising a new issue, however I have come across a new error when attempting the s5p-compress script and wondered if you had any ideas of how to over come it. I have tried everything I can think of to get around it, and I seem to get the same error no matter what product or band name I use.

Any ideas would be greatly appreciated. Thank you again for all the help previously.

python s5p-compress.py L2_data/L2__NO2___/S5P_OFFL_L2__NO2____20210310T102908_20210310T121038_17650_01_010400_20210312T035825.nc nitrogendioxide_tropospheric_column

/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rasterio/init.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py:761: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
warnings.warn(str(rio_warning.message), type(rio_warning.message))
Traceback (most recent call last):
File "s5p-compress.py", line 184, in
export_dir=EXPORT_DIR,
File "s5p-compress.py", line 38, in main
DS = rioxarray.open_rasterio(netcdf_file, chunks={"time": chunk_size})
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 774, in open_rasterio
mask_and_scale=mask_and_scale,
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 552, in load_subdatasets
default_name=subdataset.split(":")[-1].lstrip("/").replace("/", ""),
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 865, in open_rasterio
result = _prepare_dask(result, riods, filename, chunks)
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 603, in _prepare_dask
return result.chunk(chunks, name_prefix=name_prefix, token=token)
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/xarray/core/dataarray.py", line 1058, in chunk
chunks, name_prefix=name_prefix, token=token, lock=lock
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/xarray/core/dataset.py", line 1916, in chunk
"some chunks keys are not dimensions on this " "object: %s" % bad_dims
ValueError: some chunks keys are not dimensions on this object: {'time'}

[HichemOmr]

Dear Beth Wilkins,

I think the problem comes from the band name.
Please use this band name: tropospheric_NO2_column_number_density

you may also use more options (refer to the guideline) such as:
python s5p-compress.py nc_file band_name --time-resolution 1D --shp shape_file --agg-func mean

I hope it helps,

Note: by the way, can we organize a meeting for a short discussion!

cheers

[LillaBW2327]

Thank you so much for your reply @HichemOmr.

I also thought the problem was the band name as I was completely unsure what else it could be, so I ran the following script to display the names of the variables in the .nc files:

import netCDF4 as nc4
from netCDF4 import Dataset
import numpy as np
s5p_file = 'L2_data/L2__O3____/S5P_OFFL_L2__O3_____20210310T102908_20210310T121038_17650_01_020104_20210312T035820.nc'
fh = Dataset(s5p_file, mode='r')
print(fh.groups['PRODUCT'].variables.keys())

Which gave the following return:

python variables.py
odict_keys(['scanline', 'ground_pixel', 'time', 'corner', 'polynomial_exponents', 'intensity_offset_polynomial_exponents', 'layer', 'vertices', 'latitude', 'longitude', 'delta_time', 'time_utc', 'qa_value', 'nitrogendioxide_tropospheric_column', 'nitrogendioxide_tropospheric_column_precision', 'nitrogendioxide_tropospheric_column_precision_kernel', 'averaging_kernel', 'air_mass_factor_troposphere', 'air_mass_factor_total', 'tm5_tropopause_layer_index', 'tm5_constant_a', 'tm5_constant_b'])

I have tried the band name you suggested and added some of the optional functions but unfortunately got the same issue of:
anaconda3/envs/xarray/lib/python3.7/site-packages/xarray/core/dataset.py", line 1916, in chunk
"some chunks keys are not dimensions on this " "object: %s" % bad_dims
ValueError: some chunks keys are not dimensions on this object: {'time'}

I have checked that my conda environment meets the requirements.txt file (as I thought maybe the version of xarray was wrong or something similar), and have tried multiple band names and product types and always get this same weird issue. I have also tried it on a colleague's machine and they have had the same issue. Apologies if this is not an issue directly related the the s5p-tools package, I was just wondering if maybe you knew a way around it. Thank you, Beth.

[bhushanpawar1707]

Launched 4 processes

Traceback (most recent call last):
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 199, in _acquire_with_cache_info
file = self.cache[self.key]
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\lru_cache.py", line 53, in getitem
value = self.cache[key]
KeyError: [<class 'netCDF4.netCDF4.Dataset'>, ('C:\Users\pawar\s5p-tools-master\L2_data\L2__O3\S5P_NRTI_L2__O3_____20210928T081425_20210928T081925_20514_02_020201_20210928T090545.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 401, in
main(
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 221, in main
attributes = {
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 223, in
"time_coverage_start": xr.open_dataset(filename).attrs[
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\api.py", line 497, in open_dataset
backend_ds = backend.open_dataset(
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 551, in open_dataset
store = NetCDF4DataStore.open(
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 380, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 328, in init
self.format = self.ds.data_model
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 389, in ds
return self.acquire()
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4.py", line 383, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "C:\Users\pawar.conda\envs\project1\lib\contextlib.py", line 119, in enter
return next(self.gen)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self.args, **kwargs)
File "src\netCDF4_netCDF4.pyx", line 2307, in netCDF4.netCDF4.Dataset.init
File "src\netCDF4_netCDF4.pyx", line 1925, in netCDF4.netCDF4.ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:\Users\pawar\s5p-tools-master\L2_data\L2__O3\S5P_NRTI_L2__O3_____20210928T081425_20210928T081925_20514_02_020201_20210928T090545.nc'

(project1) C:\Users\pawar\s5p-tools-master>Python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)

(project1) C:\Users\pawar\s5p-tools-master>Python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)

(project1) C:\Users\pawar\s5p-tools-master>python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)

Will You please guide me on the above errors