Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow using general compression like zstd for compressing string and binary type of data #2876

Open
niyue opened this issue Sep 13, 2024 · 0 comments

Comments

@niyue
Copy link
Contributor

niyue commented Sep 13, 2024

Currently, Lance v2 uses FSST to encode string and binary data types when possible. However, compared to general compression methods like zstd, FSST may result in a lower compression ratio and may use significantly more disk space for certain types of strings. Lance v2 does allow users to specify field metadata lance:compression to request general compression, but the current implementation still attempts to apply FSST, even when the user explicitly requests another compression method.

I might be missing something, but it would be ideal if we could honor the user’s explicit choice for general compression in these cases, allowing them to bypass FSST and use the specified compression method. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant