Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix zarr folder suffix handling #3349

Merged
merged 4 commits into from
Aug 29, 2024

Conversation

jonahpearl
Copy link
Contributor

While troubleshooting for #3348, I noticed some odd behavior with zarr-based sorting analyzers. Specifically, in some parts of the code, it seemed like the user would be permitted to pass a folder name with the .zarr suffix included (ie folder=/my/favorite/analyzer/ instead of folder=/my/favorite/analyzer.zarr/). This led to two error cases when using create_sorting_analyzer if the user tried to make a new folder without the suffix:

  1. it would be created by create_zarr with the suffix, but then would fail to load directly after because the load_from_zarr function didn't add the .zarr suffix.
  2. when checking for the folder's existence when overwrite=True in the beginning of the function, it would not delete the folder (since the real folder has the .zarr extension, but the user's folder arg doesn't). But then later in SortingAnalyzer.create_zarr there is another check for whether the folder exists, and when the suffix'd folder does indeed exist, it raises an error, defeating the point of the overwrite kwarg.

This PR fixes that by:

  1. creating a single clean_zarr_folder_name in core_tools.py that will check for a .zarr suffix and add one if there isn't already
  2. cleaning the user-provided folder to ensure that if mode="zarr", the folder passed to internal funcs always has .zarr appended. Cleaning happens in:
  • create_sorting_analyzer
  • load_sorting_analyzer
  • merge_units
  • save_as
  • select_units
  • remove_units
  • [not copy, since that goes just into memory]

I also added cleaning in SortingAnalyzer.create_zarr() and SortingAnalyzer._save_or_select_or_merge() themselves which might be redundant. Hopefully I didn't miss any other entry points but I haven't extensively tested.

Obviously another solution here would be to insist that the user just put the .zarr suffix on themselves. But I like this way, as it allows for minimal change to a user's code (mode from binary_folder to zarr) and they will get the desired zarr behavior.

@samuelgarcia
Copy link
Member

merci.
This look ok.

@alejoe91 alejoe91 added the core Changes to core module label Aug 29, 2024
@samuelgarcia samuelgarcia merged commit 9238023 into SpikeInterface:main Aug 29, 2024
15 checks passed
@jonahpearl jonahpearl deleted the zarr_folder_suffix branch August 29, 2024 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Changes to core module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants