Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Balinus/patch 1 savedoc #438

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Balinus
Copy link
Contributor

@Balinus Balinus commented Sep 13, 2024

Documentation about compression #215

Copy link

codecov bot commented Sep 13, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.40%. Comparing base (59d23fc) to head (96d46c2).
Report is 43 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #438      +/-   ##
==========================================
+ Coverage   68.30%   68.40%   +0.09%     
==========================================
  Files          12       12              
  Lines        1773     1807      +34     
==========================================
+ Hits         1211     1236      +25     
- Misses        562      571       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Sep 13, 2024

Pull Request Test Coverage Report for Build 10853841802

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 68.476%

Totals Coverage Status
Change from base Build 10829126157: 0.0%
Covered Lines: 1236
Relevant Lines: 1805

💛 - Coveralls

Save a dataset to Zarr format with compression:

````@example write
using Blosc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to add Blosc to the Project toml, explicitly. Or, loaded it from whatever other package is also available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding Blosc to YAXArrays's Project.toml? or perhaps you mean YAXArrays/docs/Project.toml ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to docs 😄 .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can access the compressor within Zarr.jl directly. I will adapt the code

savedataset(ds, path="ds.zarr", driver=:zarr, compressor=compression)
nothing # hide
````

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have more than 1 compression level, and compare them both at the end, namely, what their disk sizes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, compare on my computer and report sizes in the documentation?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here, directly. Let's say a ds_level_1.zarr and ds_level4.zarr. And then query their sizes and print them.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had never use this argument, so..., also I don't know what are the defaults, and hence how different things actually are.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I see I didn't even thought that the code were actually run! We'll look into it

Copy link
Contributor Author

@Balinus Balinus Sep 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Humm, strangely, the compression does not seems to work as expected. There is little effect on Zarr folders and there is some on netcdf files...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

76M test1.nc
103M test_default.nc
75M test9.nc

84M test1.zarr
83M test_default.zarr
82M test9.zarr

Copy link
Collaborator

@lazarusA lazarusA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot! some minor comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants