ENH: Add support for _desc- entity to label different processings of a file #1315

yarikoptic · 2023-08-09T17:55:39Z

TODO outside of this PR: check/decide on possibility of organize to actually automagically assign such desc... unlikely since the space is huge...

…a file

yarikoptic · 2023-08-09T17:57:45Z

typing check fails...

typing: commands[0]> mypy dandi
dandi/keyring.py:113: error: Item "KeyringBackend" of
"Optional[KeyringBackend]" has no attribute "file_path"  [union-attr]
            if op.exists(kb.file_path):
                         ^~~~~~~~~~~~
dandi/keyring.py:113: error: Item "None" of "Optional[KeyringBackend]" has no
attribute "file_path"  [union-attr]
            if op.exists(kb.file_path):
                         ^~~~~~~~~~~~
dandi/keyring.py:115: error: Incompatible return value type (got
"Tuple[Optional[KeyringBackend], T]", expected
"Tuple[KeyringBackend, Optional[T]]")  [return-value]
                return (kb, func(kb))
                       ^~~~~~~~~~~~~~
dandi/keyring.py:115: error: Argument 1 has incompatible type
"Optional[KeyringBackend]"; expected "KeyringBackend"  [arg-type]
                return (kb, func(kb))
                                 ^~
dandi/keyring.py:1[33](https://github.com/dandi/dandi-cli/actions/runs/5812417503/job/15757766044?pr=1315#step:5:34): error: Incompatible return value type (got
"Tuple[Optional[KeyringBackend], None]", expected
"Tuple[KeyringBackend, Optional[T]]")  [return-value]
                return (kb, None)
                       ^~~~~~~~~~
dandi/tests/fixtures.py:657: error: No overload variant of "VideoWriter"
matches argument types "str", "None", "Any", "int", "Tuple[int, int]", "None" 
[call-overload]
            writer1 = cv2.VideoWriter(
                      ^
dandi/tests/fixtures.py:657: note: Possible overload variants:
dandi/tests/fixtures.py:657: note:     def __init__(self) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:660: error: Module has no attribute
"VideoWriter_fourcc"  [attr-defined]
                fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
                       ^~~~~~~~~~~~~~~~~~~~~~
dandi/tests/fixtures.py:665: error: No overload variant of "VideoWriter"
matches argument types "str", "None", "Any", "int", "Tuple[int, int]", "None" 
[call-overload]
            writer2 = cv2.VideoWriter(
                      ^
dandi/tests/fixtures.py:665: note: Possible overload variants:
dandi/tests/fixtures.py:665: note:     def __init__(self) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:668: error: Module has no attribute
"VideoWriter_fourcc"  [attr-defined]
                fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
                       ^~~~~~~~~~~~~~~~~~~~~~
Found 9 errors in 2 files (checked 78 source files)
typing: exit 1 (12.99 seconds) /home/runner/work/dandi-cli/dandi-cli> mypy dandi pid=1939
.pkg: _exit> python /opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  typing: FAIL code 1 (71.21=setup[58.21]+cmd[12.99] seconds)
  evaluation failed :( (71.[34](https://github.com/dandi/dandi-cli/actions/runs/5812417503/job/15757766044?pr=1315#step:5:35) seconds)
Error: Process completed with exit code 1.

seems to point to externals, right @jwodder ? do not fail for me locally

❯ tox -e typing
typing: install_deps> python -I -m pip install mypy types-appdirs types-python-dateutil types-requests
.pkg: install_requires> python -I -m pip install 'setuptools>=38.3.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
typing: install_package_deps> python -I -m pip install 'anys~=0.2' appdirs 'bidsschematools~=0.7.0' click click-didyoumean coverage 'dandischema~=0.8.0' duecredit 'etelemetry>=0.2.2' fasteners 'fscacher>=0.3.0' 'fsspec[http]' 'hdmf!=3.5.0' humanize 'interleave~=0.1' joblib 'keyring!=23.9.0' keyrings.alt 'nwbinspector>=0.4.28' opencv-python packaging pycryptodomex 'pydantic>=1.9.0' 'pynwb!=1.1.0,!=2.3.0,>=1.0.3' 'pyout!=0.6.0,>=0.5' pytest pytest-cov pytest-mock pytest-rerunfailures python-dateutil 'requests~=2.20' responses 'ruamel.yaml<1,>=0.15' semantic-version tenacity vcrpy zarr-checksum 'zarr~=2.10'
typing: install_package> python -I -m pip install --force-reinstall --no-deps /home/yoh/proj/dandi/dandi-cli-master/.tox/.tmp/package/1/dandi-0.55.1+33.gbe742ab9.tar.gz
typing: commands[0]> mypy dandi
Success: no issues found in 78 source files
.pkg: _exit> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
  typing: OK (8.18=setup[6.37]+cmd[1.81] seconds)
  congratulations :) (8.24 seconds)
tox -e typing  6.63s user 0.81s system 88% cpu 8.360 total

codecov · 2023-08-09T18:01:26Z

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.01% ⚠️

Comparison is base (4cde72f) 88.82% compared to head (be742ab) 88.82%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1315      +/-   ##
==========================================
- Coverage   88.82%   88.82%   -0.01%     
==========================================
  Files          76       76              
  Lines       10191    10197       +6     
==========================================
+ Hits         9052     9057       +5     
- Misses       1139     1140       +1

Flag	Coverage Δ
unittests	`88.82% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
dandi/consts.py	`98.11% <ø> (ø)`
dandi/organize.py	`83.57% <ø> (ø)`
dandi/tests/test_organize.py	`95.50% <100.00%> (+0.15%)`	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jwodder · 2023-08-09T18:32:04Z

@yarikoptic I suspect you're using an older version of mypy (v1.4.0 was released on June 20). Try again with a fresh tox environment, either by deleting .tox/ with git clean -dX or by passing -r/--recreate to the tox command.

CodyCBakerPhD · 2023-08-09T18:36:49Z

automagically assign such desc... unlikely since the space is huge...

While there are plenty of fields in NWB metadata you could pull from, they are all extensive free-text so I'd advise against.

My suggestion would be that if you can simply allow it in the dandi naming convention, we can build the tools in NeuroConv + GUIDE to allow people to set simple 1-2 word classifiers to be used for distinguishing between files

jwodder · 2023-08-09T18:46:44Z

@yarikoptic #1316 should fix the typing errors.

yarikoptic · 2023-08-09T18:58:19Z

@yarikoptic I suspect you're using an older version of mypy (v1.4.0 was released on June 20). Try again with a fresh tox environment, either by deleting .tox/ with git clean -dX or by passing -r/--recreate to the tox command.

indeed, re-creation of .tox brought them up

❯ tox -e typing
.pkg: _optional_hooks> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
typing: install_package> python -I -m pip install --force-reinstall --no-deps /home/yoh/proj/dandi/dandi-cli-master/.tox/.tmp/package/2/dandi-0.55.1+33.gbe742ab9.tar.gz
typing: commands[0]> mypy dandi
Success: no issues found in 78 source files
.pkg: _exit> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
  typing: OK (2.39=setup[2.22]+cmd[0.17] seconds)
  congratulations :) (2.42 seconds)
❯ rm -rf .tox
❯ tox -e typing
typing: install_deps> python -I -m pip install mypy types-appdirs types-python-dateutil types-requests
.pkg: install_requires> python -I -m pip install 'setuptools>=38.3.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
typing: install_package_deps> python -I -m pip install 'anys~=0.2' appdirs 'bidsschematools~=0.7.0' click click-didyoumean coverage 'dandischema~=0.8.0' duecredit 'etelemetry>=0.2.2' fasteners 'fscacher>=0.3.0' 'fsspec[http]' 'hdmf!=3.5.0' humanize 'interleave~=0.1' joblib 'keyring!=23.9.0' keyrings.alt 'nwbinspector>=0.4.28' opencv-python packaging pycryptodomex 'pydantic>=1.9.0' 'pynwb!=1.1.0,!=2.3.0,>=1.0.3' 'pyout!=0.6.0,>=0.5' pytest pytest-cov pytest-mock pytest-rerunfailures python-dateutil 'requests~=2.20' responses 'ruamel.yaml<1,>=0.15' semantic-version tenacity vcrpy zarr-checksum 'zarr~=2.10'
typing: install_package> python -I -m pip install --force-reinstall --no-deps /home/yoh/proj/dandi/dandi-cli-master/.tox/.tmp/package/1/dandi-0.55.1+33.gbe742ab9.tar.gz
typing: commands[0]> mypy dandi
dandi/keyring.py:113: error: Item "KeyringBackend" of "KeyringBackend | None" has no attribute "file_path"  [union-attr]
            if op.exists(kb.file_path):
                         ^~~~~~~~~~~~
dandi/keyring.py:113: error: Item "None" of "KeyringBackend | None" has no attribute "file_path"  [union-attr]
            if op.exists(kb.file_path):
                         ^~~~~~~~~~~~
dandi/keyring.py:115: error: Incompatible return value type (got "tuple[KeyringBackend | None, T]", expected "tuple[KeyringBackend, T | None]")  [return-value]
                return (kb, func(kb))
                       ^~~~~~~~~~~~~~
dandi/keyring.py:115: error: Argument 1 has incompatible type "KeyringBackend | None"; expected "KeyringBackend"  [arg-type]
                return (kb, func(kb))
                                 ^~
dandi/keyring.py:133: error: Incompatible return value type (got "tuple[KeyringBackend | None, None]", expected "tuple[KeyringBackend, T | None]")  [return-value]
                return (kb, None)
                       ^~~~~~~~~~
dandi/tests/fixtures.py:657: error: No overload variant of "VideoWriter" matches argument types "str", "None", "Any", "int", "tuple[int, int]", "None"  [call-overload]
            writer1 = cv2.VideoWriter(
                      ^
dandi/tests/fixtures.py:657: note: Possible overload variants:
dandi/tests/fixtures.py:657: note:     def __init__(self) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:657: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:660: error: Module has no attribute "VideoWriter_fourcc"  [attr-defined]
                fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
                       ^~~~~~~~~~~~~~~~~~~~~~
dandi/tests/fixtures.py:665: error: No overload variant of "VideoWriter" matches argument types "str", "None", "Any", "int", "tuple[int, int]", "None"  [call-overload]
            writer2 = cv2.VideoWriter(
                      ^
dandi/tests/fixtures.py:665: note: Possible overload variants:
dandi/tests/fixtures.py:665: note:     def __init__(self) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, isColor: bool = ...) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:665: note:     def __init__(self, filename: str, apiPreference: int, fourcc: int, fps: float, frameSize: Any, params: Sequence[int]) -> VideoWriter
dandi/tests/fixtures.py:668: error: Module has no attribute "VideoWriter_fourcc"  [attr-defined]
                fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
                       ^~~~~~~~~~~~~~~~~~~~~~
Found 9 errors in 2 files (checked 78 source files)
typing: exit 1 (5.34 seconds) /home/yoh/proj/dandi/dandi-cli-master> mypy dandi pid=705925
.pkg: _exit> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
  typing: FAIL code 1 (56.71=setup[51.37]+cmd[5.34] seconds)
  evaluation failed :( (56.75 seconds)
tox -e typing  25.97s user 3.92s system 52% cpu 56.844 total

github-actions · 2023-08-11T18:06:06Z

🚀 PR was released in 0.56.0 🚀

ENH: Add support for _desc- entity to label different processings of …

be742ab

…a file

yarikoptic requested a review from jwodder August 9, 2023 17:55

yarikoptic mentioned this pull request Aug 9, 2023

question about naming when a spike sorting is included as a separate NWB file #1314

Open

jwodder approved these changes Aug 9, 2023

View reviewed changes

yarikoptic mentioned this pull request Aug 9, 2023

Fix typing errors due to keyring and cv externals changes. #1316

Merged

yarikoptic added minor Increment the minor version when merged release Create a release when this pr is merged labels Aug 11, 2023

yarikoptic merged commit 292a2ff into master Aug 11, 2023
21 of 24 checks passed

yarikoptic deleted the enh-desc branch August 11, 2023 18:04

github-actions bot added the released label Aug 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Add support for _desc- entity to label different processings of a file #1315

ENH: Add support for _desc- entity to label different processings of a file #1315

yarikoptic commented Aug 9, 2023

yarikoptic commented Aug 9, 2023

codecov bot commented Aug 9, 2023 •

edited

Loading

jwodder commented Aug 9, 2023

CodyCBakerPhD commented Aug 9, 2023

jwodder commented Aug 9, 2023

yarikoptic commented Aug 9, 2023

github-actions bot commented Aug 11, 2023

ENH: Add support for _desc- entity to label different processings of a file #1315

ENH: Add support for _desc- entity to label different processings of a file #1315

Conversation

yarikoptic commented Aug 9, 2023

yarikoptic commented Aug 9, 2023

codecov bot commented Aug 9, 2023 • edited Loading

Codecov Report

jwodder commented Aug 9, 2023

CodyCBakerPhD commented Aug 9, 2023

jwodder commented Aug 9, 2023

yarikoptic commented Aug 9, 2023

github-actions bot commented Aug 11, 2023

codecov bot commented Aug 9, 2023 •

edited

Loading