You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, Lance v2 uses FSST to encode string and binary data types when possible. However, compared to general compression methods like zstd, FSST may result in a lower compression ratio and may use significantly more disk space for certain types of strings. Lance v2 does allow users to specify field metadata lance:compression to request general compression, but the current implementation still attempts to apply FSST, even when the user explicitly requests another compression method.
I might be missing something, but it would be ideal if we could honor the user’s explicit choice for general compression in these cases, allowing them to bypass FSST and use the specified compression method. Thanks!
The text was updated successfully, but these errors were encountered:
Currently, Lance v2 uses FSST to encode
string
andbinary
data types when possible. However, compared to general compression methods likezstd
, FSST may result in a lower compression ratio and may use significantly more disk space for certain types of strings. Lance v2 does allow users to specify field metadatalance:compression
to request general compression, but the current implementation still attempts to apply FSST, even when the user explicitly requests another compression method.I might be missing something, but it would be ideal if we could honor the user’s explicit choice for general compression in these cases, allowing them to bypass FSST and use the specified compression method. Thanks!
The text was updated successfully, but these errors were encountered: