Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integer compression in 2.1 #2857

Open
Tracked by #2856
westonpace opened this issue Sep 11, 2024 · 0 comments
Open
Tracked by #2856

Integer compression in 2.1 #2857

westonpace opened this issue Sep 11, 2024 · 0 comments

Comments

@westonpace
Copy link
Contributor

Integer compression is very important. It's useful for integral types, decimal types, and temporal types. It can also benefit string / binary types as well as they store their offsets into integers. It can even benefit other encodings (such as structural encodings) that might need integer metadata arrays.

The state of the art appears to be a combination of bit packing, frame of reference, and delta encoding. We should implement these features in version 2.1 of Lance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant