feat: teach FSSTArray to compress the offsets of its codes #928
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
More subtle than I expected.
DeltaArray is a sequence of chunks. All chunks except the last must be "full", i.e. containing 1,024
values. The last chunk may contain as few as one value and is encoded differently from the rest.
In this PR, I introduced an "offset" and a "limit". Together they enable logical/lazy slicing while
preserving full chunks for later decompression. The offset is a value, less than 1024, which offsets
into the first chunk. The limit is either `None` or less than 1024. `None` represents no limit which
allows callers to avoid computing the length of the last chunk [1]. Internally, the limit is
converted to a "right-offset": `trailing_garbage` which is sliced away when decompression happens.
[1] Which is, a bit annoyingly, this:
```
match deltas.len() % 1024 {
0 => 1024,
n => n
}
```
The codes of an FSSTArray are a vector of binary-strings of one byte codes or an escape code followed by a data. The offsets, unexpectedly, grow quite large, increasing the file size (for example, the TPC-H l_comment column with this PR is 78% the byte size of itself on `develop`). Delta encoding notably decreases the size but also inflates the compression time, seemingly proportional to the space savings (TPC-H l_comment compresses in 111% of the time on `develop`).
danking
changed the base branch from
dk/teach-delta-array-scalar-at-and-slice
to
develop
danking
added
benchmark
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Vortex bytes_at
| Benchmark suite | Current: 1e83368 | Previous: a7fd730 | Ratio |
|---|---|---|---|
bytes_at/array_data |
585.8659272970784 ns (1.5227584104723064) |
589.1412744092991 ns (1.1526929581398235) |
0.99 |
bytes_at/array_view |
873.1899630379073 ns (0.5722827191509055) |
865.5491107046423 ns (2.5108739632320862) |
1.01 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Random Access
| Benchmark suite | Current: 1e83368 | Previous: a7fd730 | Ratio |
|---|---|---|---|
random-access/vortex-tokio-local-disk |
1166726.1763205817 ns (10416.304236468393) |
1124465.0316005207 ns (4016.6545367563376) |
1.04 |
random-access/vortex-local-fs |
1298903.0162465351 ns (3863.4341095724376) |
1279703.9340791185 ns (4555.701691351482) |
1.02 |
random-access/parquet-tokio-local-disk |
193060658.93333334 ns (2397485.060833335) |
190959983.93333334 ns (2287012.7966666818) |
1.01 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
DataFusion
| Benchmark suite | Current: 1e83368 | Previous: a7fd730 | Ratio |
|---|---|---|---|
arrow/planning |
816396.3268275453 ns (911.1846863506944) |
823203.1065386933 ns (3069.4351141784573) |
0.99 |
arrow/exec |
1761952.0484163098 ns (1024.8611673369305) |
1790910.9321507337 ns (6372.738233479555) |
0.98 |
vortex-pushdown-compressed/planning |
512725.46147507033 ns (1028.7630345559737) |
512864.84533388744 ns (729.7778903637372) |
1.00 |
vortex-pushdown-compressed/exec |
3097796.0282352935 ns (5902.188749999041) |
3401013.542666667 ns (36988.230441666674) |
0.91 |
vortex-pushdown-uncompressed/planning |
511932.8747658665 ns (521.9356939914869) |
514132.22840157093 ns (1018.4282293085707) |
1.00 |
vortex-pushdown-uncompressed/exec |
3328843.18875 ns (5731.960867187474) |
3450584.587999999 ns (13142.269375000382) |
0.96 |
vortex-nopushdown-compressed/planning |
825662.1877206456 ns (754.178369455738) |
837026.7734598288 ns (2956.2392458583345) |
0.99 |
vortex-nopushdown-compressed/exec |
14813977.845 ns (72573.81356250029) |
9074584.881666668 ns (54816.66668749787) |
1.63 |
vortex-nopushdown-uncompressed/planning |
822575.3948570885 ns (568.2211421735119) |
828200.6579283918 ns (864.2747718397295) |
0.99 |
vortex-nopushdown-uncompressed/exec |
1781906.5512991853 ns (2077.257371983258) |
1808399.0196487666 ns (17152.327237995923) |
0.99 |
This comment was automatically generated by workflow using github-action-benchmark.
robert3005
reviewed
Sep 25, 2024
Comment on lines
+99
to
+103
| let codes_offsets_delta = DeltaCompressor.compress( | ||
| &codes_varbin.offsets(), | ||
| like.as_ref().and_then(|l| l.child(1).cloned()), | ||
| ctx.auxiliary("offsets"), | ||
| )?; |
There was a problem hiding this comment.
you could completely delegate to the compressor and it would pick the best variant for it
There was a problem hiding this comment.
Vortex Compression
| Benchmark suite | Current: 1e83368 | Previous: a7fd730 | Ratio |
|---|---|---|---|
Yellow Taxi Trip Data Compression Time/taxi compression |
2535042344 ns (6563432.25) |
2505564549 ns (2459766.649999857) |
1.01 |
Yellow Taxi Trip Data Compression Time/taxi compression throughput |
470808924 bytes |
470808924 bytes |
1 |
Yellow Taxi Trip Data Vortex-to-ParquetZstd Ratio/taxi |
0.961091974357628 ratio |
0.9623810200038339 ratio |
1.00 |
Yellow Taxi Trip Data Vortex-to-ParquetUncompressed Ratio/taxi |
0.6169442332004692 ratio |
0.6177716974796194 ratio |
1.00 |
Yellow Taxi Trip Data Compression Ratio/taxi |
0.10807140520556488 ratio |
0.10834444166143291 ratio |
1.00 |
Yellow Taxi Trip Data Compression Size/taxi |
50880982 bytes |
51009530 bytes |
1.00 |
Public BI Compression Time/AirlineSentiment compression |
411955.70731624484 ns (475.388599130325) |
409513.7859096733 ns (870.394846893003) |
1.01 |
Public BI Compression Time/AirlineSentiment compression throughput |
2020 bytes |
2020 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/AirlineSentiment |
6.4672897196261685 ratio |
6.4672897196261685 ratio |
1 |
Public BI Vortex-to-ParquetUncompressed Ratio/AirlineSentiment |
4.398305084745763 ratio |
4.398305084745763 ratio |
1 |
Public BI Compression Ratio/AirlineSentiment |
0.6207920792079208 ratio |
0.6207920792079208 ratio |
1 |
Public BI Compression Size/AirlineSentiment |
1254 bytes |
1254 bytes |
1 |
Public BI Compression Time/Arade compression |
3314027174.6 ns (11252932.491250038) |
3242124033.9 ns (1469835.3262500763) |
1.02 |
Public BI Compression Time/Arade compression throughput |
787023760 bytes |
787023760 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/Arade |
0.4758240782134243 ratio |
0.4758240782134243 ratio |
1 |
Public BI Vortex-to-ParquetUncompressed Ratio/Arade |
0.4247114758619328 ratio |
0.4247114758619328 ratio |
1 |
Public BI Compression Ratio/Arade |
0.17931668543272442 ratio |
0.17931668543272442 ratio |
1 |
Public BI Compression Size/Arade |
141126492 bytes |
141126492 bytes |
1 |
Public BI Compression Time/Bimbo compression |
24202284532.8 ns (26893770.85874939) |
23429189446.6 ns (8843205.906251907) |
1.03 |
Public BI Compression Time/Bimbo compression throughput |
7121333608 bytes |
7121333608 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/Bimbo |
1.355213884858197 ratio |
1.3546765695893752 ratio |
1.00 |
Public BI Vortex-to-ParquetUncompressed Ratio/Bimbo |
0.9188800729817758 ratio |
0.918515755364501 ratio |
1.00 |
Public BI Compression Ratio/Bimbo |
0.06753251588434782 ratio |
0.06753188018880972 ratio |
1.00 |
Public BI Compression Size/Bimbo |
480921575 bytes |
480917048 bytes |
1.00 |
Public BI Compression Time/CMSprovider compression |
13985696957.2 ns (47424057.09874916) |
12849421993.4 ns (7553227.1000003815) |
1.09 |
Public BI Compression Time/CMSprovider compression throughput |
5149123964 bytes |
5149123964 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/CMSprovider |
1.1808158482573146 ratio |
1.2444998371178397 ratio |
0.95 |
Public BI Vortex-to-ParquetUncompressed Ratio/CMSprovider |
0.7624444012379048 ratio |
0.803564700247157 ratio |
0.95 |
Public BI Compression Ratio/CMSprovider |
0.17253849260794374 ratio |
0.1818402680817649 ratio |
0.95 |
Public BI Compression Size/CMSprovider |
888422087 bytes |
936318082 bytes |
0.95 |
Public BI Compression Time/Euro2016 compression |
2101161777.9 ns (2332703.649999976) |
2038043814.2 ns (5023478.056250095) |
1.03 |
Public BI Compression Time/Euro2016 compression throughput |
393253221 bytes |
393253221 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/Euro2016 |
1.4117935554073797 ratio |
1.4636567880878564 ratio |
0.96 |
Public BI Vortex-to-ParquetUncompressed Ratio/Euro2016 |
0.5989933841012823 ratio |
0.6209978288267344 ratio |
0.96 |
Public BI Compression Ratio/Euro2016 |
0.4217737049380709 ratio |
0.4323139517273019 ratio |
0.98 |
Public BI Compression Size/Euro2016 |
165863868 bytes |
170008854 bytes |
0.98 |
Public BI Compression Time/Food compression |
1184854513.9 ns (8566569.215000033) |
1129654884.4 ns (1213048.1687500477) |
1.05 |
Public BI Compression Time/Food compression throughput |
332718229 bytes |
332718229 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/Food |
1.2681811382598833 ratio |
1.268188204730334 ratio |
1.00 |
Public BI Vortex-to-ParquetUncompressed Ratio/Food |
0.7170605154521362 ratio |
0.7170645110067071 ratio |
1.00 |
Public BI Compression Ratio/Food |
0.13348678289580582 ratio |
0.13348678289580582 ratio |
1 |
Public BI Compression Size/Food |
44413486 bytes |
44413486 bytes |
1 |
Public BI Compression Time/HashTags compression |
3062019095.8 ns (5868010.57249999) |
2880580864.5 ns (1121821.0999999046) |
1.06 |
Public BI Compression Time/HashTags compression throughput |
804495592 bytes |
804495592 bytes |
1 |
Public BI Vortex-to-ParquetZstd Ratio/HashTags |
1.5381589750706188 ratio |
1.663730097809073 ratio |
0.92 |
Public BI Vortex-to-ParquetUncompressed Ratio/HashTags |
0.43730163358702806 ratio |
0.4730017517119196 ratio |
0.92 |
Public BI Compression Ratio/HashTags |
0.2522608016974691 ratio |
0.26456875726424117 ratio |
0.95 |
Public BI Compression Size/HashTags |
202942703 bytes |
212844399 bytes |
0.95 |
TPC-H l_comment Compression Time/chunked-without-fsst compression |
193325328.47027782 ns (781157.5644722581) |
183662042.41434523 ns (1194779.002205342) |
1.05 |
TPC-H l_comment Compression Time/chunked-without-fsst compression throughput |
183010921 bytes |
183010921 bytes |
1 |
TPC-H l_comment Vortex-to-ParquetZstd Ratio/chunked-without-fsst |
3.21553977937666 ratio |
3.2154507270869335 ratio |
1.00 |
TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/chunked-without-fsst |
0.9983780671585054 ratio |
0.9983798790604546 ratio |
1.00 |
TPC-H l_comment Compression Ratio/chunked-without-fsst |
0.999965750677797 ratio |
0.999965750677797 ratio |
1 |
TPC-H l_comment Compression Size/chunked-without-fsst |
183004653 bytes |
183004653 bytes |
1 |
TPC-H l_comment Compression Time/chunked-with-fsst compression |
1243315103.45 ns (4157993.0749999285) |
1120506207.3 ns (181774.27625000477) |
1.11 |
TPC-H l_comment Compression Time/chunked-with-fsst compression throughput |
183010921 bytes |
183010921 bytes |
1 |
TPC-H l_comment Vortex-to-ParquetZstd Ratio/chunked-with-fsst |
1.164405864127871 ratio |
1.5073288115458823 ratio |
0.77 |
TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/chunked-with-fsst |
0.3615309888162447 ratio |
0.46801735877908546 ratio |
0.77 |
TPC-H l_comment Compression Ratio/chunked-with-fsst |
0.359850044140262 ratio |
0.4435869704191041 ratio |
0.81 |
TPC-H l_comment Compression Size/chunked-with-fsst |
65856488 bytes |
81181260 bytes |
0.81 |
TPC-H l_comment Compression Time/canonical-with-fsst compression |
1254989313.5 ns (3428257.326874852) |
1121884658.8 ns (369269.71062505245) |
1.12 |
TPC-H l_comment Compression Time/canonical-with-fsst compression throughput |
183010937 bytes |
183010937 bytes |
1 |
TPC-H l_comment Vortex-to-ParquetZstd Ratio/canonical-with-fsst |
1.1644118974029094 ratio |
1.5073326767703459 ratio |
0.77 |
TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/canonical-with-fsst |
0.3615309888162447 ratio |
0.46801735877908546 ratio |
0.77 |
TPC-H l_comment Compression Ratio/canonical-with-fsst |
0.3598420568711694 ratio |
0.44357897582918776 ratio |
0.81 |
TPC-H l_comment Compression Size/canonical-with-fsst |
65855032 bytes |
81179804 bytes |
0.81 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
TPC-H
| Benchmark suite | Current: 1e83368 | Previous: a7fd730 | Ratio |
|---|---|---|---|
tpch_q1/vortex-in-memory-no-pushdown |
486676401.6 ns (3033088.028125018) |
470778811.05 ns (2570272) |
1.03 |
tpch_q1/vortex-in-memory-pushdown |
528718836.3 ns (4837168.682500005) |
518780781.6 ns (5362983.314999998) |
1.02 |
tpch_q1/arrow |
463155686.05 ns (1832451.598750025) |
447889731.15 ns (2284537.650000006) |
1.03 |
tpch_q1/parquet |
678221553.7 ns (5487649.199999988) |
655604780.7 ns (4704334.399999976) |
1.03 |
tpch_q1/vortex-file-compressed |
635143012.3 ns (6159323.427500069) |
629753524.3 ns (1311501.75) |
1.01 |
tpch_q1/vortex-file-uncompressed |
547945819.2 ns (3061128.5962499976) |
540280392.4 ns (1017531.6525000334) |
1.01 |
tpch_q2/vortex-in-memory-no-pushdown |
134526092.70099205 ns (2132978.5582137927) |
123911864.99023807 ns (368851.1829762012) |
1.09 |
tpch_q2/vortex-in-memory-pushdown |
132395858.95218253 ns (1427707.633442454) |
122613522.92837301 ns (406964.34402777255) |
1.08 |
tpch_q2/arrow |
130474810.92019841 ns (1796973.8515431657) |
122530415.23420635 ns (601704.0055952445) |
1.06 |
tpch_q2/parquet |
168296671.46543652 ns (2881244.3589811474) |
156986252.63829365 ns (459657.51143850386) |
1.07 |
tpch_q2/vortex-file-compressed |
166875531.12976193 ns (2117504.1516666412) |
156515107.638254 ns (1286721.699801594) |
1.07 |
tpch_q2/vortex-file-uncompressed |
169567581.43357146 ns (3646344.8704077303) |
159559362.1426984 ns (1235152.2134563476) |
1.06 |
tpch_q3/vortex-in-memory-no-pushdown |
173026340.05079365 ns (2644913.3544543535) |
153736156.77726188 ns (769150.5334538668) |
1.13 |
tpch_q3/vortex-in-memory-pushdown |
201941063.86666667 ns (4876790.866666675) |
187970880.70000002 ns (2774801.115416661) |
1.07 |
tpch_q3/arrow |
162706695.43134922 ns (3589595.0693253726) |
145546463.70623016 ns (834212.5704692304) |
1.12 |
tpch_q3/parquet |
358244624.1 ns (4004581.099999994) |
332846368.4 ns (1284458.1600000262) |
1.08 |
tpch_q3/vortex-file-compressed |
345670853 ns (3157362.5612499714) |
330735168.65 ns (2381750.1775000095) |
1.05 |
tpch_q3/vortex-file-uncompressed |
298276777.25 ns (3941371.7306250036) |
280147171.45 ns (1200292.0087499917) |
1.06 |
tpch_q4/vortex-in-memory-no-pushdown |
121752580.91468254 ns (4989560.37238989) |
112648346.40623017 ns (982913.5290545523) |
1.08 |
tpch_q4/vortex-in-memory-pushdown |
149305860.98761904 ns (1235452.48541072) |
147131511.45805556 ns (520651.66444444656) |
1.01 |
tpch_q4/arrow |
111142607.45876984 ns (1904201.9311750978) |
104389292.01166667 ns (609267.8640208319) |
1.06 |
tpch_q4/parquet |
230110525.9333333 ns (2518717.227916658) |
216992847.7666667 ns (1173126.8516666591) |
1.06 |
tpch_q4/vortex-file-compressed |
301903123.9 ns (3115694.6024999917) |
291661199.95 ns (960266.8518750072) |
1.04 |
tpch_q4/vortex-file-uncompressed |
240723587.9333333 ns (1533686.8833333254) |
229143403.66666666 ns (1067383.4587499797) |
1.05 |
tpch_q5/vortex-in-memory-no-pushdown |
333960431.8 ns (5717726.366249979) |
303406513.05 ns (4344806.557499975) |
1.10 |
tpch_q5/vortex-in-memory-pushdown |
337860097.5 ns (4025976.974999994) |
320111346.4 ns (3271559.688749999) |
1.06 |
tpch_q5/arrow |
326939585.95 ns (3825615.3350000083) |
287290302.3 ns (2605396.9318749905) |
1.14 |
tpch_q5/parquet |
489817908.15 ns (3568462.0287499726) |
450382744.1 ns (5991275.376874983) |
1.09 |
tpch_q5/vortex-file-compressed |
377973850.25 ns (5239321.433124989) |
353369393.1 ns (3282339.3012500107) |
1.07 |
tpch_q5/vortex-file-uncompressed |
374296837.35 ns (4037524.525000006) |
342814463.1 ns (2269862.7231250107) |
1.09 |
tpch_q6/vortex-in-memory-no-pushdown |
45669599.0615344 ns (423533.7050277777) |
41145302.50825396 ns (264241.5752361119) |
1.11 |
tpch_q6/vortex-in-memory-pushdown |
96077233.22190475 ns (338094.76272619516) |
89005534.20527777 ns (884844.0098993033) |
1.08 |
tpch_q6/arrow |
37496243.37851852 ns (112567.25119907781) |
35280898.30301587 ns (288395.20619047433) |
1.06 |
tpch_q6/parquet |
156971278.63861114 ns (921648.0832083225) |
151786552.90293652 ns (1148404.0190872997) |
1.03 |
tpch_q6/vortex-file-compressed |
66733103.76081349 ns (392672.3282564394) |
66446906.92785714 ns (270419.90361160785) |
1.00 |
tpch_q6/vortex-file-uncompressed |
183849527.70000002 ns (1009665.0754166692) |
182235586.5236508 ns (1227112.9938888848) |
1.01 |
tpch_q7/vortex-in-memory-no-pushdown |
627983155.8 ns (18805181.733749986) |
556362295.2 ns (4778926.372500002) |
1.13 |
tpch_q7/vortex-in-memory-pushdown |
675707021.3 ns (13775709.805000007) |
614219882.5 ns (3438564.019999981) |
1.10 |
tpch_q7/arrow |
614837865.1 ns (13115771.671249986) |
556031432.5 ns (4067206.651250005) |
1.11 |
tpch_q7/parquet |
771309099.2 ns (13484820.592499971) |
718007743.5 ns (5807857.472499967) |
1.07 |
tpch_q7/vortex-file-compressed |
767632780.9 ns (7018559.050000012) |
727618997.8 ns (6774538.868750036) |
1.05 |
tpch_q7/vortex-file-uncompressed |
778637560.7 ns (23449853.64624995) |
703464836.1 ns (3966395.058749974) |
1.11 |
tpch_q8/vortex-in-memory-no-pushdown |
235593667.03333336 ns (2429266.8224999607) |
218345345.7333333 ns (734491.9391666502) |
1.08 |
tpch_q8/vortex-in-memory-pushdown |
252909480.7 ns (4095837.232500002) |
234827427.63333336 ns (1735342.7062500268) |
1.08 |
tpch_q8/arrow |
226856376.66666666 ns (2969246.8833333254) |
206458989.3 ns (448134.4833333492) |
1.10 |
tpch_q8/parquet |
514021551 ns (8806512.900000006) |
478596306.15 ns (2889889.903125018) |
1.07 |
tpch_q8/vortex-file-compressed |
305180522.25 ns (2222781.1212500036) |
286440556.25 ns (1136457.5149999857) |
1.07 |
tpch_q8/vortex-file-uncompressed |
298876794.25 ns (2651275.4650000036) |
286617141.9 ns (2189069.3587499857) |
1.04 |
tpch_q9/vortex-in-memory-no-pushdown |
444780745.6 ns (7516365.246874988) |
411318121.45 ns (5785188.35375002) |
1.08 |
tpch_q9/vortex-in-memory-pushdown |
454565564.05 ns (6812080.059374988) |
403319522.4 ns (1622460.431250006) |
1.13 |
tpch_q9/arrow |
422176829.4 ns (3722110.734375) |
382454852.05 ns (847782.0006249845) |
1.10 |
tpch_q9/parquet |
738623681.9 ns (7053013.946250021) |
684319507.9 ns (4741953.976249993) |
1.08 |
tpch_q9/vortex-file-compressed |
495139711.4 ns (7183686.936249971) |
453439742 ns (912271.8243749738) |
1.09 |
tpch_q9/vortex-file-uncompressed |
471760436.15 ns (5447565.847500026) |
447005220.6 ns (1331298.0912499726) |
1.06 |
tpch_q10/vortex-in-memory-no-pushdown |
247968954.76666665 ns (3245362.0424999744) |
225938863.9 ns (1832863.8333332986) |
1.10 |
tpch_q10/vortex-in-memory-pushdown |
276996559.7 ns (3608921.112500012) |
255467335.1 ns (2450173.324999988) |
1.08 |
tpch_q10/arrow |
239649088.76666665 ns (3896788.3541666865) |
224986890.7 ns (1646367.0337500274) |
1.07 |
tpch_q10/parquet |
500350580.6 ns (3281057.8912500143) |
474759648.1 ns (2474133.849999994) |
1.05 |
tpch_q10/vortex-file-compressed |
472399289.7 ns (1719038.6550000012) |
452473163.5 ns (990036.9931249917) |
1.04 |
tpch_q10/vortex-file-uncompressed |
369456934.45 ns (3307039.9506249726) |
355951521.05 ns (815710) |
1.04 |
tpch_q11/vortex-in-memory-no-pushdown |
189411767.66666666 ns (2851901.605000004) |
174378287.5119841 ns (291298.1973412633) |
1.09 |
tpch_q11/vortex-in-memory-pushdown |
191235492 ns (2896888.59708333) |
175921071.72027776 ns (158039.80420833826) |
1.09 |
tpch_q11/arrow |
189255372.16666666 ns (2190449.1974999905) |
172947300.2111905 ns (259850.16488093138) |
1.09 |
tpch_q11/parquet |
204920570.5 ns (5143651.3054166585) |
180698830.6666667 ns (396665.1158333421) |
1.13 |
tpch_q11/vortex-file-compressed |
262744132.4 ns (4162734.2462499887) |
225651668.2 ns (417650.1537500024) |
1.16 |
tpch_q11/vortex-file-uncompressed |
246086196.2 ns (3466907.6116666645) |
224485407.43333334 ns (695158.2775000036) |
1.10 |
tpch_q12/vortex-in-memory-no-pushdown |
208546899.4 ns (1135535.8587499857) |
199949568.4333333 ns (216992.74541668594) |
1.04 |
tpch_q12/vortex-in-memory-pushdown |
256641259.05 ns (1802419.559375003) |
247251831.33333334 ns (216615.39999999106) |
1.04 |
tpch_q12/arrow |
175161327.04646823 ns (632077.0189523846) |
165716337.61130953 ns (210387.7949181497) |
1.06 |
tpch_q12/parquet |
376094840.35 ns (2891123.471875012) |
354963622 ns (1742009.1943750083) |
1.06 |
tpch_q12/vortex-file-compressed |
660338655.6 ns (2624277.571249962) |
646608024.4 ns (888956.8650000095) |
1.02 |
tpch_q12/vortex-file-uncompressed |
362392038.45 ns (2702433.0818749964) |
352656401.75 ns (1914967.1831249893) |
1.03 |
tpch_q13/vortex-in-memory-no-pushdown |
218668529.2 ns (6982680.653750002) |
162449208.60079366 ns (3761308.810436502) |
1.35 |
tpch_q13/vortex-in-memory-pushdown |
218981846.8333333 ns (4355724.849999964) |
161397514.6402381 ns (2289703.17285417) |
1.36 |
tpch_q13/arrow |
213412056.1666667 ns (5527431.312916666) |
161859092.6793254 ns (4039361.621840775) |
1.32 |
tpch_q13/parquet |
363108811.25 ns (7285689.152500033) |
300414832.6 ns (1623244.150000006) |
1.21 |
tpch_q13/vortex-file-compressed |
230512224.06666666 ns (2039674.3537499756) |
204082228.5 ns (1010463.9425000101) |
1.13 |
tpch_q13/vortex-file-uncompressed |
210238691.3 ns (2065105.6616666615) |
196188812.96666667 ns (2201124.4166666716) |
1.07 |
tpch_q14/vortex-in-memory-no-pushdown |
48822197.41555555 ns (562757.8301388919) |
46187656.0361508 ns (452375.7624245994) |
1.06 |
tpch_q14/vortex-in-memory-pushdown |
92512352.11686507 ns (1240077.4677852169) |
86577988.0137103 ns (702697.1248906255) |
1.07 |
tpch_q14/arrow |
42095553.391044974 ns (601318.7526742704) |
39815975.24281746 ns (242407.20347619057) |
1.06 |
tpch_q14/parquet |
230695039.8333333 ns (831832.6087500155) |
223447063 ns (732257.2524999976) |
1.03 |
tpch_q14/vortex-file-compressed |
124548375.85523811 ns (1002615.6822976172) |
122679410.6998016 ns (904560.2744662687) |
1.02 |
tpch_q14/vortex-file-uncompressed |
160906608.4722619 ns (2515115.885833323) |
156947637.98825398 ns (439542.25064484775) |
1.03 |
tpch_q15/vortex-in-memory-no-pushdown |
85103412.77313492 ns (1704902.3391577452) |
76070838.32097222 ns (656435.4381319359) |
1.12 |
tpch_q15/vortex-in-memory-pushdown |
122728197.01686506 ns (1361172.3844647706) |
121912388.94861111 ns (2306124.6742743105) |
1.01 |
tpch_q15/arrow |
71682672.51434524 ns (1588147.1932120547) |
66639598.062678576 ns (371171.91783257946) |
1.08 |
tpch_q15/parquet |
303218110 ns (1490826.3431249857) |
293979971.8 ns (1256768.783125013) |
1.03 |
tpch_q15/vortex-file-compressed |
235112837.9 ns (2458658.9804166704) |
220365691.93333334 ns (1586283.7183333337) |
1.07 |
tpch_q15/vortex-file-uncompressed |
324023539.7 ns (2476958.7287499905) |
313248672.75 ns (1304307.5587499738) |
1.03 |
tpch_q16/vortex-in-memory-no-pushdown |
112764683.323373 ns (845598.8895858005) |
106665388.64789681 ns (508544.1264484152) |
1.06 |
tpch_q16/vortex-in-memory-pushdown |
130419796.82650793 ns (1224013.196176596) |
124446532.89305556 ns (469445.67838542163) |
1.05 |
tpch_q16/arrow |
111124937.03265874 ns (1013088.3808913603) |
105214140.25710317 ns (88176.1345238015) |
1.06 |
tpch_q16/parquet |
129719640.0245635 ns (1036443.6787991002) |
122221498.5122619 ns (374059.6328422725) |
1.06 |
tpch_q16/vortex-file-compressed |
143005066.8265873 ns (2320224.4915525615) |
140114230.44404763 ns (730503.6269047707) |
1.02 |
tpch_q16/vortex-file-uncompressed |
137988669.0570238 ns (968949.5656220317) |
133967057.03119047 ns (761671.2557410672) |
1.03 |
tpch_q17/vortex-in-memory-no-pushdown |
746326646.2 ns (13949250.933750033) |
588852715.3 ns (15265776.800000012) |
1.27 |
tpch_q17/vortex-in-memory-pushdown |
837494493.5 ns (23465137.502499998) |
664425256.2 ns (13721128.076250017) |
1.26 |
tpch_q17/arrow |
744519699.5 ns (24582523.600000024) |
572163055.6 ns (6032225.367500007) |
1.30 |
tpch_q17/parquet |
630465135.8 ns (13256209.349999964) |
589894670.9 ns (1265204.7662500143) |
1.07 |
tpch_q17/vortex-file-compressed |
647370360.4 ns (14495588.721249938) |
613135777.1 ns (2043911.626249969) |
1.06 |
tpch_q17/vortex-file-uncompressed |
685960932.5 ns (19803557.035000026) |
607969922.3 ns (2370896.2325000167) |
1.13 |
tpch_q18/vortex-in-memory-no-pushdown |
1231850509.3 ns (23999830.762500048) |
1026736344.1 ns (7462663.150000036) |
1.20 |
tpch_q18/vortex-in-memory-pushdown |
1243666053.3 ns (20721423.59499991) |
1036578945.9 ns (18223705.142500043) |
1.20 |
tpch_q18/arrow |
1227289263.7 ns (20176164.316249967) |
992603041.9 ns (4561204.769999981) |
1.24 |
tpch_q18/parquet |
1391527507 ns (17331542.274999857) |
1190800765 ns (4771371.134999871) |
1.17 |
tpch_q18/vortex-file-compressed |
1266815337 ns (17713219.100000024) |
1066628218.2 ns (5466607.400000036) |
1.19 |
tpch_q18/vortex-file-uncompressed |
1216031179.4 ns (12754006.706249952) |
1041264120.5 ns (7907268.0737499595) |
1.17 |
tpch_q19/vortex-in-memory-no-pushdown |
168375021.1947619 ns (641083.62787202) |
162171607.0890476 ns (236975.6219047606) |
1.04 |
tpch_q19/vortex-in-memory-pushdown |
264116747.6 ns (1043123.7412499934) |
239775955.1333333 ns (1148010.6275000125) |
1.10 |
tpch_q19/arrow |
156311392.2960714 ns (365512.02821725607) |
150228233.46448413 ns (449639.8877777755) |
1.04 |
tpch_q19/parquet |
495258182.05 ns (2710943.3881250024) |
472818045.7 ns (2131794.0774999857) |
1.05 |
tpch_q19/vortex-file-compressed |
621487376.6 ns (2842383.207499981) |
640209783.2 ns (1281863.0262500048) |
0.97 |
tpch_q19/vortex-file-uncompressed |
340718687.2 ns (3221593.060624987) |
326651702.25 ns (1052329.9299999774) |
1.04 |
tpch_q20/vortex-in-memory-no-pushdown |
293906711.65 ns (6104608.886875004) |
271593039.03333336 ns (5503360.348333359) |
1.08 |
tpch_q20/vortex-in-memory-pushdown |
308012218.3 ns (5615887.07249999) |
279644235.75 ns (1367248.646874994) |
1.10 |
tpch_q20/arrow |
288720519.35 ns (4542979.405000001) |
252453238.43333334 ns (1425655.5999999642) |
1.14 |
tpch_q20/parquet |
400180310.85 ns (4270792.393124998) |
372016936.05 ns (1734170.3387500048) |
1.08 |
tpch_q20/vortex-file-compressed |
397672733.9 ns (6159384.120000035) |
375373899.6 ns (2591469.505000025) |
1.06 |
tpch_q20/vortex-file-uncompressed |
428651545.6 ns (4665854.46312499) |
405133354.45 ns (1444914.6168750226) |
1.06 |
tpch_q21/vortex-in-memory-no-pushdown |
920398005.8 ns (9409937.808749974) |
874031875.5 ns (9584810.050000012) |
1.05 |
tpch_q21/vortex-in-memory-pushdown |
973199649.9 ns (14857608.191250026) |
910428293.7 ns (3489386.2162500024) |
1.07 |
tpch_q21/arrow |
898570011.7 ns (13880417.116249979) |
843367204.1 ns (2119223.1575000286) |
1.07 |
tpch_q21/parquet |
1082260238.6 ns (11440665.319999993) |
999693809.1 ns (5988897.466250062) |
1.08 |
tpch_q21/vortex-file-compressed |
1305006633.7 ns (9235890.20749998) |
1254936501.1 ns (6398013.162500024) |
1.04 |
tpch_q21/vortex-file-uncompressed |
1173378195.6 ns (7741148.606250048) |
1112135742.7 ns (4256241.850000024) |
1.06 |
tpch_q22/vortex-in-memory-no-pushdown |
71988099.03833333 ns (861272.8572291657) |
68539415.30855158 ns (302082.6290734112) |
1.05 |
tpch_q22/vortex-in-memory-pushdown |
73300402.71831349 ns (1363264.3537301645) |
69386406.87198414 ns (159689.68498264998) |
1.06 |
tpch_q22/arrow |
70990104.79011904 ns (898929.6035446525) |
66870334.98736111 ns (313804.3846666664) |
1.06 |
tpch_q22/parquet |
102438676.47531746 ns (1412465.961993046) |
96079838.88257936 ns (532226.812982142) |
1.07 |
tpch_q22/vortex-file-compressed |
111418620.43777779 ns (3409803.7024444193) |
104202847.0836111 ns (397156.4162986055) |
1.07 |
tpch_q22/vortex-file-uncompressed |
110471861.50496033 ns (1422562.5577182472) |
104078185.30797617 ns (324778.7296785861) |
1.06 |
This comment was automatically generated by workflow using github-action-benchmark.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The codes of an FSSTArray are a vector of binary-strings of one byte codes or an escape code followed by a data. The offsets, unexpectedly, grow quite large, increasing the file size (for example, the TPC-H l_comment column with this PR is 78% the byte size of itself on
develop). Delta encoding notably decreases the size but also inflates the compression time, seemingly proportional to the space savings (TPC-H l_comment compresses in 111% of the time ondevelop).