ZSTD codec

[vthacker]

Summary

Describe the goal of this PR. Mention any related Issue numbers.

Requirements

• I've read and understood the Contributing Guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.

[bryanlb]

I looked a bit more into this, and I'm not convinced that ZSTD is the direction we want to go. The discussion from OpenSearch here specifically raises some concerns:

One of those reasons is the hard dependency on native compiled code, which often leads to portability issues due to glibc differences.

There's also currently issues with memory leaks, and aside from that the performance (especially at query time) is a mixed bag. If we were already running zlib this may be more interesting, but we currently use the default of BEST_SPEED.

apache/lucene#9784 (comment)

Codec	Indexing time (ms)	Disk usage (MB)	Retrieval time per 10k docs (ms)
BEST_SPEED (LZ4 with small blocks)	35383	90.175	190.17524
BEST_COMPRESSION (vanilla zlib, DEFLATE level=6)	76671	58.682	1910.42106
BEST_COMPRESSION (Cloudflare zlib, DEFLATE level=6)	54791	58.601	1395.53593
ZSTD dict (level=1)	24687	63.324	928.73997
ZSTD dict (level=2)	24934	63.722	977.29911
ZSTD dict (level=3)	28285	62.072	938.10886
ZSTD dict (level=4)	37863	60.427	969.18655
ZSTD dict (level=5)	45479	59.317	941.20922
ZSTD dict (level=6)	57842	58.481	881.69049
ZSTD dict (level=7)	65796	58.107	886.42249

I think this is something we can table for now, and potentially revisit in the future.