feat: update IPC format to hold buffer_index #903

a10y · 2024-09-20T21:04:28Z

Fixes #786

Updates IPC format, removing the Array.has_buffer boolean field and replacing it with a buffer_index. The buffer_index for each array is an optional uint64 that will hold the index into the Arc<[Buffer]> containing the buffer of the given array, or null if the array does not have a buffer.

Additional changes:

Moves children into an Arc<[Array]> from Vec<Array> to improve cloning performance
Uses scalar_at_unchecked in ChunkedArray::chunk to lookup chunk offsets, avoiding validity check since by construction, chunk offsets must be non-nullable

github-actions

DataFusion

Benchmark suite	Current: `8b218cd`	Previous: `e9f0d4d`	Ratio
`arrow/planning`	`829267.632566971` ns (`932.3446902072174`)	`825701.8204083241` ns (`1289.499651635706`)	`1.00`
`arrow/exec`	`1794665.3175348146` ns (`2319.453966166242`)	`1765393.828450285` ns (`1709.4757793694735`)	`1.02`
`vortex-pushdown-compressed/planning`	`519402.18819745025` ns (`1015.7659451537475`)	`513577.851936208` ns (`1791.075647192367`)	`1.01`
`vortex-pushdown-compressed/exec`	`3292236.196875` ns (`12539.926843749825`)	`3137998.252352941` ns (`11435.878455883125`)	`1.05`
`vortex-pushdown-uncompressed/planning`	`521328.8906794807` ns (`1012.7205875158252`)	`513444.1068847204` ns (`558.9450257514254`)	`1.02`
`vortex-pushdown-uncompressed/exec`	`3009626.126470587` ns (`4767.322727941675`)	`3354325.63875` ns (`19272.99874218763`)	`0.90`
`vortex-nopushdown-compressed/planning`	`847607.4296193782` ns (`2150.7791104926728`)	`842389.7683694257` ns (`682.9176903121988`)	`1.01`
`vortex-nopushdown-compressed/exec`	`14153752.775` ns (`122935.4439687496`)	`9196856.046666665` ns (`372627.80604166724`)	`1.54`
`vortex-nopushdown-uncompressed/planning`	`837808.8329706627` ns (`905.7927670691861`)	`837868.8191581981` ns (`2622.0283910088474`)	`1.00`
`vortex-nopushdown-uncompressed/exec`	`1823753.0551414334` ns (`4871.627666815766`)	`1809878.5197882229` ns (`6570.107307738275`)	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex bytes_at

Benchmark suite	Current: `8b218cd`	Previous: `e9f0d4d`	Ratio
`bytes_at/array_data`	`614.1059981219782` ns (`0.32029886467728375`)	`623.017561289198` ns (`0.30176197246072434`)	`0.99`
`bytes_at/array_view`	`904.9343055699625` ns (`2.346675060546488`)	`998.7505087067391` ns (`0.2729134184216946`)	`0.91`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Random Access

Benchmark suite	Current: `8b218cd`	Previous: `e9f0d4d`	Ratio
`random-access/vortex-tokio-local-disk`	`1129288.568338388` ns (`10239.786555711296`)	`1251597.7092830988` ns (`4887.412258077995`)	`0.90`
`random-access/vortex-local-fs`	`1266425.971887876` ns (`4234.64614577312`)	`1412827.7215840626` ns (`5789.054523073137`)	`0.90`
`random-access/parquet-tokio-local-disk`	`195850616.26666665` ns (`1924836.5037499964`)	`191705484.26666665` ns (`3667323.243749976`)	`1.02`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex Compression

Benchmark suite	Current: `8b218cd`	Previous: `e9f0d4d`	Ratio
`Yellow Taxi Trip Data Compression Time/taxi compression`	`2514404306.2` ns (`4959842.900000095`)	`2565510302.7` ns (`7490440.25`)	`0.98`
`Yellow Taxi Trip Data Compression Time/taxi compression throughput`	`470808924` bytes	`470808924` bytes	`1`
`Yellow Taxi Trip Data Vortex-to-ParquetZstd Ratio/taxi`	`0.9585447652590474` ratio	`0.9536893981178375` ratio	`1.01`
`Yellow Taxi Trip Data Vortex-to-ParquetUncompressed Ratio/taxi`	`0.6153091285423793` ratio	`0.6121923708981979` ratio	`1.01`
`Yellow Taxi Trip Data Compression Ratio/taxi`	`0.10811712651394008` ratio	`0.10757489380129082` ratio	`1.01`
`Yellow Taxi Trip Data Compression Size/taxi`	`50902508` bytes	`50647220` bytes	`1.01`
`Public BI Compression Time/AirlineSentiment compression`	`407097.69234779535` ns (`1816.6425907142693`)	`413876.7175462057` ns (`2321.4872500271304`)	`0.98`
`Public BI Compression Time/AirlineSentiment compression throughput`	`2020` bytes	`2020` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/AirlineSentiment`	`6.4672897196261685` ratio	`6.400830737279335` ratio	`1.01`
`Public BI Vortex-to-ParquetUncompressed Ratio/AirlineSentiment`	`4.398305084745763` ratio	`4.353107344632768` ratio	`1.01`
`Public BI Compression Ratio/AirlineSentiment`	`0.6207920792079208` ratio	`0.6207920792079208` ratio	`1`
`Public BI Compression Size/AirlineSentiment`	`1254` bytes	`1254` bytes	`1`
`Public BI Compression Time/Arade compression`	`3147729346.6` ns (`4259457.645000219`)	`3210142661.9` ns (`14799204.643749952`)	`0.98`
`Public BI Compression Time/Arade compression throughput`	`787023760` bytes	`787023760` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/Arade`	`0.4925282243224146` ratio	`0.4923085928627726` ratio	`1.00`
`Public BI Vortex-to-ParquetUncompressed Ratio/Arade`	`0.4396212773448676` ratio	`0.43942523850269616` ratio	`1.00`
`Public BI Compression Ratio/Arade`	`0.18587409101854815` ratio	`0.18587409101854815` ratio	`1`
`Public BI Compression Size/Arade`	`146287326` bytes	`146287326` bytes	`1`
`Public BI Compression Time/Bimbo compression`	`22483561880.2` ns (`22350357.37999916`)	`23173051974.1` ns (`34524850.168748856`)	`0.97`
`Public BI Compression Time/Bimbo compression throughput`	`7121333608` bytes	`7121333608` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/Bimbo`	`1.2932583668173148` ratio	`1.3005279554405174` ratio	`0.99`
`Public BI Vortex-to-ParquetUncompressed Ratio/Bimbo`	`0.8768721718120011` ratio	`0.8818011946026248` ratio	`0.99`
`Public BI Compression Ratio/Bimbo`	`0.06428802682206824` ratio	`0.06465913863138316` ratio	`0.99`
`Public BI Compression Size/Bimbo`	`457816486` bytes	`460459297` bytes	`0.99`
`Public BI Compression Time/CMSprovider compression`	`13365042234.2` ns (`16551070.901249886`)	`13598568220` ns (`76919069.61874962`)	`0.98`
`Public BI Compression Time/CMSprovider compression throughput`	`5149123964` bytes	`5149123964` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/CMSprovider`	`1.2034613251149986` ratio	`1.202885760410426` ratio	`1.00`
`Public BI Vortex-to-ParquetUncompressed Ratio/CMSprovider`	`0.7770664247049723` ratio	`0.7766947866657309` ratio	`1.00`
`Public BI Compression Ratio/CMSprovider`	`0.1758624856055223` ratio	`0.17584668466529077` ratio	`1.00`
`Public BI Compression Size/CMSprovider`	`905537739` bytes	`905456378` bytes	`1.00`
`Public BI Compression Time/Euro2016 compression`	`2173422484.8` ns (`1851417.5025000572`)	`2221942324.2` ns (`7139216.276250124`)	`0.98`
`Public BI Compression Time/Euro2016 compression throughput`	`393253221` bytes	`393253221` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/Euro2016`	`1.464575159686757` ratio	`1.4642817760282572` ratio	`1.00`
`Public BI Vortex-to-ParquetUncompressed Ratio/Euro2016`	`0.6213874739768918` ratio	`0.6212629976540103` ratio	`1.00`
`Public BI Compression Ratio/Euro2016`	`0.43303186065957233` ratio	`0.4329904115394391` ratio	`1.00`
`Public BI Compression Size/Euro2016`	`170291174` bytes	`170274874` bytes	`1.00`
`Public BI Compression Time/Food compression`	`1084788231.7` ns (`1319475.1499999762`)	`1093010464.5` ns (`3201702.0299999714`)	`0.99`
`Public BI Compression Time/Food compression throughput`	`332718229` bytes	`332718229` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/Food`	`1.231458457945433` ratio	`1.2307288448714029` ratio	`1.00`
`Public BI Vortex-to-ParquetUncompressed Ratio/Food`	`0.6962966172355176` ratio	`0.6958840762260664` ratio	`1.00`
`Public BI Compression Ratio/Food`	`0.1300739611715113` ratio	`0.1300739611715113` ratio	`1`
`Public BI Compression Size/Food`	`43277978` bytes	`43277978` bytes	`1`
`Public BI Compression Time/HashTags compression`	`2776534025.3` ns (`2390727.892500162`)	`2960966405.3` ns (`22318037.29249978`)	`0.94`
`Public BI Compression Time/HashTags compression throughput`	`804495592` bytes	`804495592` bytes	`1`
`Public BI Vortex-to-ParquetZstd Ratio/HashTags`	`1.652484879855117` ratio	`1.9578739076847234` ratio	`0.84`
`Public BI Vortex-to-ParquetUncompressed Ratio/HashTags`	`0.46980471404480767` ratio	`0.5566274176234868` ratio	`0.84`
`Public BI Compression Ratio/HashTags`	`0.262578655620527` ratio	`0.3136872140873085` ratio	`0.84`
`Public BI Compression Size/HashTags`	`211243371` bytes	`252359981` bytes	`0.84`
`TPC-H l_comment Compression Time/chunked-without-fsst compression`	`193777109.4576984` ns (`399648.271208331`)	`195376090.90275794` ns (`1464397.9258940965`)	`0.99`
`TPC-H l_comment Compression Time/chunked-without-fsst compression throughput`	`183010921` bytes	`183010921` bytes	`1`
`TPC-H l_comment Vortex-to-ParquetZstd Ratio/chunked-without-fsst`	`3.2155978396821525` ratio	`3.215479570024902` ratio	`1.00`
`TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/chunked-without-fsst`	`0.9983822594666735` ratio	`0.9983765137938084` ratio	`1.00`
`TPC-H l_comment Compression Ratio/chunked-without-fsst`	`0.999965750677797` ratio	`0.999965750677797` ratio	`1`
`TPC-H l_comment Compression Size/chunked-without-fsst`	`183004653` bytes	`183004653` bytes	`1`
`TPC-H l_comment Compression Time/chunked-with-fsst compression`	`1128768563.4` ns (`1893173.875`)	`1147762309.3` ns (`7355471.25`)	`0.98`
`TPC-H l_comment Compression Time/chunked-with-fsst compression throughput`	`183010921` bytes	`183010921` bytes	`1`
`TPC-H l_comment Vortex-to-ParquetZstd Ratio/chunked-with-fsst`	`1.5074855260018034` ratio	`1.505925935696729` ratio	`1.00`
`TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/chunked-with-fsst`	`0.4680457198316051` ratio	`0.4675760031966042` ratio	`1.00`
`TPC-H l_comment Compression Ratio/chunked-with-fsst`	`0.44361324207531855` ratio	`0.4431715908363742` ratio	`1.00`
`TPC-H l_comment Compression Size/chunked-with-fsst`	`81186068` bytes	`81105241` bytes	`1.00`
`TPC-H l_comment Compression Time/canonical-with-fsst compression`	`1131559915.2` ns (`2432056.294374943`)	`1136178299.05` ns (`2516500.098749995`)	`1.00`
`TPC-H l_comment Compression Time/canonical-with-fsst compression throughput`	`183010937` bytes	`183010937` bytes	`1`
`TPC-H l_comment Vortex-to-ParquetZstd Ratio/canonical-with-fsst`	`1.5074891270207975` ratio	`1.5059199579926357` ratio	`1.00`
`TPC-H l_comment Vortex-to-ParquetUncompressed Ratio/canonical-with-fsst`	`0.4680456815449925` ratio	`0.46757606694437` ratio	`1.00`
`TPC-H l_comment Compression Ratio/canonical-with-fsst`	`0.4436052474831053` ratio	`0.44316359628277296` ratio	`1.00`
`TPC-H l_comment Compression Size/canonical-with-fsst`	`81184612` bytes	`81103785` bytes	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

TPC-H

Benchmark suite	Current: `8b218cd`	Previous: `e9f0d4d`	Ratio
`tpch_q1/vortex-in-memory-no-pushdown`	`471707075.4` ns (`1323100.849999994`)	`475582603.45` ns (`3783747.914375007`)	`0.99`
`tpch_q1/vortex-in-memory-pushdown`	`534469459.9` ns (`4069500.050000012`)	`519153426.9` ns (`3427080.650000006`)	`1.03`
`tpch_q1/arrow`	`461277485.05` ns (`1042702.099999994`)	`454139804.75` ns (`3941862.046875`)	`1.02`
`tpch_q1/parquet`	`667700530.6` ns (`1728788.1000000238`)	`666138707.2` ns (`4604618.349999964`)	`1.00`
`tpch_q1/vortex-file-compressed`	`631589224` ns (`1452918.512499988`)	`618225520.5` ns (`1095452.905000031`)	`1.02`
`tpch_q1/vortex-file-uncompressed`	`651520275.8` ns (`3858980.9000000358`)	`637380852.8` ns (`1795846.0012500286`)	`1.02`
`tpch_q2/vortex-in-memory-no-pushdown`	`129185540.92884922` ns (`227633.20043651015`)	`126594453.19531746` ns (`1257125.2075158805`)	`1.02`
`tpch_q2/vortex-in-memory-pushdown`	`130946094.38031745` ns (`433825.2492857203`)	`124615912.37087302` ns (`618017.7889622971`)	`1.05`
`tpch_q2/arrow`	`125867110.09301586` ns (`305459.4042619243`)	`121663239.45940478` ns (`612838.5226294696`)	`1.03`
`tpch_q2/parquet`	`161779481.01777777` ns (`1365669.1311111152`)	`155001286.72123015` ns (`716643.8772544563`)	`1.04`
`tpch_q2/vortex-file-compressed`	`161801003.7954762` ns (`738479.2510714233`)	`155607858.28234127` ns (`1172999.5673199296`)	`1.04`
`tpch_q2/vortex-file-uncompressed`	`174479998.77011907` ns (`891205.1134523749`)	`168487142.68646827` ns (`981420.7672261894`)	`1.04`
`tpch_q3/vortex-in-memory-no-pushdown`	`162123817.42599204` ns (`1137449.0164563507`)	`161384588.19591272` ns (`1438657.0834841281`)	`1.00`
`tpch_q3/vortex-in-memory-pushdown`	`193530063.1333333` ns (`718396.9358333498`)	`186352815.9666667` ns (`3337630.543750018`)	`1.04`
`tpch_q3/arrow`	`149143524.14595237` ns (`567796.136592269`)	`151053987.50349206` ns (`2211112.9914533943`)	`0.99`
`tpch_q3/parquet`	`355222002.75` ns (`2163172.212500006`)	`345100903.5` ns (`2074620.873124987`)	`1.03`
`tpch_q3/vortex-file-compressed`	`439771522.7` ns (`1170943.9375`)	`332404152.1` ns (`2563114.775000006`)	`1.32`
`tpch_q3/vortex-file-uncompressed`	`417968310.05` ns (`2229729.879999995`)	`419079941.45` ns (`5252075.125`)	`1.00`
`tpch_q4/vortex-in-memory-no-pushdown`	`113738953.56190476` ns (`364753.030952394`)	`115168309.13515873` ns (`893902.0150704384`)	`0.99`
`tpch_q4/vortex-in-memory-pushdown`	`144852983.2745635` ns (`1203138.4917276949`)	`140615654.52007937` ns (`1084460.0865704566`)	`1.03`
`tpch_q4/arrow`	`107646708.20480159` ns (`1102301.8320967332`)	`105345828.67444444` ns (`1102880.0569444448`)	`1.02`
`tpch_q4/parquet`	`223795885.73333335` ns (`2142268.5900000036`)	`219069491.4` ns (`1532087.6666666716`)	`1.02`
`tpch_q4/vortex-file-compressed`	`408268434.7` ns (`1319668.103125006`)	`297953993.55` ns (`1526327.9231249988`)	`1.37`
`tpch_q4/vortex-file-uncompressed`	`323372009.7` ns (`2580215.1387500167`)	`314399917.45` ns (`1184616.75`)	`1.03`
`tpch_q5/vortex-in-memory-no-pushdown`	`308252935.85` ns (`1308969.7881250083`)	`304344033.85` ns (`2428757.51562503`)	`1.01`
`tpch_q5/vortex-in-memory-pushdown`	`329008663.75` ns (`2298489.700000018`)	`314360791.55` ns (`4983605.589375019`)	`1.05`
`tpch_q5/arrow`	`304477295.95` ns (`5144526.159999996`)	`288516744.3` ns (`1343121.582499981`)	`1.06`
`tpch_q5/parquet`	`456304967.1` ns (`7073018.954375029`)	`455705995.35` ns (`5398815.520624995`)	`1.00`
`tpch_q5/vortex-file-compressed`	`356044938.25` ns (`7286734.76000002`)	`346569164.25` ns (`2355901.1837500036`)	`1.03`
`tpch_q5/vortex-file-uncompressed`	`399499729.7` ns (`6631116.388125002`)	`383076741.7` ns (`1831304.3899999857`)	`1.04`
`tpch_q6/vortex-in-memory-no-pushdown`	`45131187.477513224` ns (`289178.1063492`)	`45279148.06875` ns (`522295.53854166344`)	`1.00`
`tpch_q6/vortex-in-memory-pushdown`	`96792044.56281745` ns (`461891.8780555576`)	`91079606.18765871` ns (`377650.4468452409`)	`1.06`
`tpch_q6/arrow`	`37079728.57701059` ns (`100811.15891766548`)	`36767709.04085978` ns (`349040.47075396776`)	`1.01`
`tpch_q6/parquet`	`155920970.2517857` ns (`873436.4606027007`)	`150374242.47750002` ns (`630247.1586666703`)	`1.04`
`tpch_q6/vortex-file-compressed`	`67166271.08845238` ns (`224378.10538541526`)	`67443685.00115079` ns (`253587.21793649346`)	`1.00`
`tpch_q6/vortex-file-uncompressed`	`260422451.35` ns (`2216718.2787500024`)	`258053084.75` ns (`4003751.423124999`)	`1.01`
`tpch_q7/vortex-in-memory-no-pushdown`	`577673983.7` ns (`5512869.150000036`)	`570632484.9` ns (`5082521.22875005`)	`1.01`
`tpch_q7/vortex-in-memory-pushdown`	`637044063.1` ns (`9715808.231250048`)	`605499508.7` ns (`2675134.9812499285`)	`1.05`
`tpch_q7/arrow`	`586022046.1` ns (`5238834.178749979`)	`564747865.7` ns (`7661455.738749981`)	`1.04`
`tpch_q7/parquet`	`727531513` ns (`6755212.600000024`)	`736889502.6` ns (`11244084.636250079`)	`0.99`
`tpch_q7/vortex-file-compressed`	`792738045.2` ns (`4201141.550000012`)	`724628415` ns (`4628239.398749948`)	`1.09`
`tpch_q7/vortex-file-uncompressed`	`840377092.1` ns (`5025833.475000024`)	`850439664.7` ns (`8186705.263749957`)	`0.99`
`tpch_q8/vortex-in-memory-no-pushdown`	`225527581.4333333` ns (`1672657.2275000066`)	`224210908.46666664` ns (`1940488.5012500137`)	`1.01`
`tpch_q8/vortex-in-memory-pushdown`	`240570266.3666667` ns (`1521649.0487499684`)	`234273868.4333333` ns (`1487392.9737499952`)	`1.03`
`tpch_q8/arrow`	`219540025` ns (`1323640.9333333522`)	`213268609.8333333` ns (`1450955.8974999934`)	`1.03`
`tpch_q8/parquet`	`504682024.8` ns (`3865086.224999994`)	`483860526.15` ns (`4706825.607499987`)	`1.04`
`tpch_q8/vortex-file-compressed`	`296959157.9` ns (`1753279.599999994`)	`280902380.6` ns (`1537650.3250000179`)	`1.06`
`tpch_q8/vortex-file-uncompressed`	`347195873.25` ns (`4202326.954374999`)	`331386151.05` ns (`1565970.373124987`)	`1.05`
`tpch_q9/vortex-in-memory-no-pushdown`	`436864114.35` ns (`4662379.471249998`)	`416917862.25` ns (`6238695.507499993`)	`1.05`
`tpch_q9/vortex-in-memory-pushdown`	`439964858` ns (`5643944.425000012`)	`436655486.95` ns (`2832524.220625013`)	`1.01`
`tpch_q9/arrow`	`422825380.35` ns (`3088770.2543750107`)	`404472196.35` ns (`4574980.21312499`)	`1.05`
`tpch_q9/parquet`	`717830450.2` ns (`3073720.7900000215`)	`699062400.8` ns (`4201221.707499981`)	`1.03`
`tpch_q9/vortex-file-compressed`	`471009229.9` ns (`6186281.158749968`)	`450592078.8` ns (`3991240.2487499714`)	`1.05`
`tpch_q9/vortex-file-uncompressed`	`509059935.2` ns (`5957559.199999988`)	`494334372.7` ns (`2369181.963749975`)	`1.03`
`tpch_q10/vortex-in-memory-no-pushdown`	`239102194.73333335` ns (`2093786.049999997`)	`235850111.5333333` ns (`3241469.666666642`)	`1.01`
`tpch_q10/vortex-in-memory-pushdown`	`274522865.75` ns (`2782118.974999994`)	`270201122.6` ns (`2187319.966874987`)	`1.02`
`tpch_q10/arrow`	`231911000.36666664` ns (`1517213.7166666985`)	`225453042.06666666` ns (`410294.16666667163`)	`1.03`
`tpch_q10/parquet`	`497563336.5` ns (`3479403.2162500024`)	`503787196.3` ns (`3763961.3650000095`)	`0.99`
`tpch_q10/vortex-file-compressed`	`508457572.2` ns (`1463795.7124999762`)	`464905922.9` ns (`1734032.9506250024`)	`1.09`
`tpch_q10/vortex-file-uncompressed`	`446167238.6` ns (`1897509.7962499857`)	`453911906.9` ns (`3148366.7381249964`)	`0.98`
`tpch_q11/vortex-in-memory-no-pushdown`	`188424977.9` ns (`855563.90625`)	`182839175.36666667` ns (`1555652.8833333105`)	`1.03`
`tpch_q11/vortex-in-memory-pushdown`	`184185695.89999998` ns (`1096939.801666692`)	`182979582.13333336` ns (`726159.2445833236`)	`1.01`
`tpch_q11/arrow`	`187282557.5` ns (`1980180.6395833641`)	`182776918.93333334` ns (`1769248.9412500113`)	`1.02`
`tpch_q11/parquet`	`199951556.4` ns (`886202.8108333349`)	`195460976.96666667` ns (`1890309.4274999946`)	`1.02`
`tpch_q11/vortex-file-compressed`	`243471360.6` ns (`2640375.448333338`)	`236473626.23333335` ns (`2090046.5679166615`)	`1.03`
`tpch_q11/vortex-file-uncompressed`	`246415271.9666667` ns (`3610229.769999981`)	`242011671.8666667` ns (`2197470.1049999595`)	`1.02`
`tpch_q12/vortex-in-memory-no-pushdown`	`205301400.90000004` ns (`635363.0333333313`)	`206744408.8333333` ns (`520718.11500000954`)	`0.99`
`tpch_q12/vortex-in-memory-pushdown`	`255099800.55` ns (`207178.1081250012`)	`251244839.2` ns (`814755.3206250072`)	`1.02`
`tpch_q12/arrow`	`171119429.13785714` ns (`841701.4955952168`)	`171396699.58543652` ns (`582803.6527306587`)	`1.00`
`tpch_q12/parquet`	`364557641.65` ns (`1825798.0750000179`)	`370099689.65` ns (`1826948.5268749893`)	`0.99`
`tpch_q12/vortex-file-compressed`	`641650607.3` ns (`1654656.1087500453`)	`650971299.1` ns (`3685168.221250057`)	`0.99`
`tpch_q12/vortex-file-uncompressed`	`438213402.7` ns (`1552766.675000012`)	`448664891.35` ns (`3362179.909375012`)	`0.98`
`tpch_q13/vortex-in-memory-no-pushdown`	`186962496.0176984` ns (`6593240.104999989`)	`192466256.23333332` ns (`4684763.956250012`)	`0.97`
`tpch_q13/vortex-in-memory-pushdown`	`190526804.84067458` ns (`2684321.4384821206`)	`189632057.06666666` ns (`4571021.471249983`)	`1.00`
`tpch_q13/arrow`	`158403175.19083333` ns (`422164.8762500137`)	`174351007.09043652` ns (`3218298.724990085`)	`0.91`
`tpch_q13/parquet`	`332897265.15` ns (`3431706.741874993`)	`335844360.2` ns (`7486460.664375007`)	`0.99`
`tpch_q13/vortex-file-compressed`	`214868791.19999996` ns (`2379809.5070833564`)	`212512806.66666666` ns (`1459966.8504166752`)	`1.01`
`tpch_q13/vortex-file-uncompressed`	`208692719.29999998` ns (`2799323.399999991`)	`215685321.5333333` ns (`2239093.730000004`)	`0.97`
`tpch_q14/vortex-in-memory-no-pushdown`	`47278659.288611114` ns (`242699.7304548584`)	`48986369.54668651` ns (`359631.9436686523`)	`0.97`
`tpch_q14/vortex-in-memory-pushdown`	`86850768.52607143` ns (`493092.69166667014`)	`88846345.49115078` ns (`732372.2313244119`)	`0.98`
`tpch_q14/arrow`	`39693658.86988095` ns (`207717.82381398603`)	`40500726.781402126` ns (`482062.82695204765`)	`0.98`
`tpch_q14/parquet`	`222940799.63333336` ns (`900421.0120833516`)	`227998490.3` ns (`1751107.8441666812`)	`0.98`
`tpch_q14/vortex-file-compressed`	`124213698.67416665` ns (`632731.9050000235`)	`123897309.43972221` ns (`312811.2287499979`)	`1.00`
`tpch_q14/vortex-file-uncompressed`	`197741853.86666667` ns (`1902542.4333333373`)	`198012926.6` ns (`895139.7050000131`)	`1.00`
`tpch_q15/vortex-in-memory-no-pushdown`	`76193201.85236111` ns (`777147.5748437494`)	`80594131.4802381` ns (`573637.9865476191`)	`0.95`
`tpch_q15/vortex-in-memory-pushdown`	`119407818.68309525` ns (`351372.5677410662`)	`121972979.41440475` ns (`550489.7561607137`)	`0.98`
`tpch_q15/arrow`	`66970152.03299602` ns (`502740.85996032134`)	`67604786.42954364` ns (`1331823.7332363576`)	`0.99`
`tpch_q15/parquet`	`293258444.4` ns (`716222.2668749988`)	`296551956.55` ns (`1848727.8662500083`)	`0.99`
`tpch_q15/vortex-file-compressed`	`222787498.7333333` ns (`1585318.8404166847`)	`226525464.1` ns (`1230926.840833336`)	`0.98`
`tpch_q15/vortex-file-uncompressed`	`365164566.2` ns (`4854641.575000018`)	`373431511.1` ns (`3965713.059375018`)	`0.98`
`tpch_q16/vortex-in-memory-no-pushdown`	`107783336.96452382` ns (`325256.7999374941`)	`107233105.84496032` ns (`269234.1732961312`)	`1.01`
`tpch_q16/vortex-in-memory-pushdown`	`125802556.22900793` ns (`192145.20848065615`)	`124996999.75777778` ns (`371460.0520416647`)	`1.01`
`tpch_q16/arrow`	`106600991.17503968` ns (`250412.66493304074`)	`108136738.00317462` ns (`1110430.5338095352`)	`0.99`
`tpch_q16/parquet`	`125609470.27761905` ns (`701204.0148006082`)	`124526377.21234128` ns (`834823.3945714235`)	`1.01`
`tpch_q16/vortex-file-compressed`	`139143807.25150794` ns (`436264.3668641001`)	`139846315.32539684` ns (`521824.42453970015`)	`0.99`
`tpch_q16/vortex-file-uncompressed`	`143874551.62305558` ns (`617872.6779166758`)	`143046456.2713492` ns (`350812.36463293433`)	`1.01`
`tpch_q17/vortex-in-memory-no-pushdown`	`617613983.1` ns (`6410859.38500005`)	`635140829.1` ns (`16196233.89499998`)	`0.97`
`tpch_q17/vortex-in-memory-pushdown`	`719225299.1` ns (`10916850.152499974`)	`661327958.2` ns (`8604574.002499998`)	`1.09`
`tpch_q17/arrow`	`588480931.8` ns (`19477798.649999976`)	`587866213.2` ns (`14823626.262499988`)	`1.00`
`tpch_q17/parquet`	`594152185.5` ns (`2049323.5812499523`)	`593131370.6` ns (`3488574.323750019`)	`1.00`
`tpch_q17/vortex-file-compressed`	`631348467.6` ns (`2515070.9037500024`)	`625965890.8` ns (`8886165.256249964`)	`1.01`
`tpch_q17/vortex-file-uncompressed`	`708144390.4` ns (`4070224.126249969`)	`687319330.9` ns (`5306300.899999976`)	`1.03`
`tpch_q18/vortex-in-memory-no-pushdown`	`1098813549` ns (`13493437.138749957`)	`1044217123` ns (`14600676.165000021`)	`1.05`
`tpch_q18/vortex-in-memory-pushdown`	`1078608750.3` ns (`11805371.814999938`)	`1061940335.9` ns (`7713772.983749986`)	`1.02`
`tpch_q18/arrow`	`1056165632.9` ns (`6266130.037500024`)	`1046690314.4` ns (`13240314.49999988`)	`1.01`
`tpch_q18/parquet`	`1232074399.4` ns (`5885991.49000001`)	`1220386242.4` ns (`17619962.05000007`)	`1.01`
`tpch_q18/vortex-file-compressed`	`1108497558.2` ns (`8010387.780000091`)	`1094192305.8` ns (`12714726.550000072`)	`1.01`
`tpch_q18/vortex-file-uncompressed`	`1178297865.6` ns (`5662645.301249981`)	`1183892638.3` ns (`12547012.899999976`)	`1.00`
`tpch_q19/vortex-in-memory-no-pushdown`	`164461442.44285715` ns (`507093.3917708397`)	`167708523.55079365` ns (`207872.60238096118`)	`0.98`
`tpch_q19/vortex-in-memory-pushdown`	`253415278.85` ns (`770891.7912500054`)	`246913352.76666665` ns (`1181537.149999991`)	`1.03`
`tpch_q19/arrow`	`150834980.81456348` ns (`211517.20560020208`)	`152907504.48908728` ns (`531873.4004563391`)	`0.99`
`tpch_q19/parquet`	`479695344.85` ns (`2502255.535624981`)	`483715749.2` ns (`1402571.5837500095`)	`0.99`
`tpch_q19/vortex-file-compressed`	`628627221.1` ns (`921412.3774999976`)	`626076814.6` ns (`2322301.473750055`)	`1.00`
`tpch_q19/vortex-file-uncompressed`	`426739993.8` ns (`1092873.0187499821`)	`426220590.35` ns (`3039137.8712500036`)	`1.00`
`tpch_q20/vortex-in-memory-no-pushdown`	`241611018.19999996` ns (`1385412.8679166585`)	`256124067.95` ns (`3743115.1568749994`)	`0.94`
`tpch_q20/vortex-in-memory-pushdown`	`258952019.2` ns (`750763.924999997`)	`279430977` ns (`4484692.425624996`)	`0.93`
`tpch_q20/arrow`	`231090019.9` ns (`595909.6770833284`)	`246777714.25` ns (`3929205.075000003`)	`0.94`
`tpch_q20/parquet`	`353436718.1` ns (`1623593.5618750155`)	`367613776.5` ns (`2965610.1075000167`)	`0.96`
`tpch_q20/vortex-file-compressed`	`341941403.35` ns (`1816261.2293750048`)	`369426996.5` ns (`1963579.5974999666`)	`0.93`
`tpch_q20/vortex-file-uncompressed`	`445797532.2` ns (`1577799.0343749821`)	`464413811.15` ns (`3691025.413749993`)	`0.96`
`tpch_q21/vortex-in-memory-no-pushdown`	`822908068.9` ns (`1729277.9762499928`)	`868169883` ns (`9030092.935000002`)	`0.95`
`tpch_q21/vortex-in-memory-pushdown`	`870158662` ns (`1004624.3300000429`)	`913158928.7` ns (`12412399.774999976`)	`0.95`
`tpch_q21/arrow`	`804227945.8` ns (`852637.7637500167`)	`839458196.6` ns (`6607976.405000031`)	`0.96`
`tpch_q21/parquet`	`959977400.3` ns (`2960985.038749993`)	`1007310504.3` ns (`10821411.507499993`)	`0.95`
`tpch_q21/vortex-file-compressed`	`1443552450.4` ns (`4646041.191249967`)	`1246545538` ns (`4285743.129999876`)	`1.16`
`tpch_q21/vortex-file-uncompressed`	`1316799855.6` ns (`7294458.5500000715`)	`1340905244.3` ns (`8834080.647499919`)	`0.98`
`tpch_q22/vortex-in-memory-no-pushdown`	`69445863.614127` ns (`339487.46042410284`)	`71001125.11301586` ns (`378351.91348809004`)	`0.98`
`tpch_q22/vortex-in-memory-pushdown`	`68776161.11972222` ns (`100030.13783680648`)	`71704457.77876985` ns (`414706.1549181491`)	`0.96`
`tpch_q22/arrow`	`66972165.31823413` ns (`131565.01843253523`)	`68476025.22970238` ns (`292803.0053980574`)	`0.98`
`tpch_q22/parquet`	`94429539.38277778` ns (`280784.9552013874`)	`98217972.13853174` ns (`614093.5969226137`)	`0.96`
`tpch_q22/vortex-file-compressed`	`100836950.7475` ns (`518505.2166666612`)	`105992979.25567462` ns (`2366776.521212809`)	`0.95`
`tpch_q22/vortex-file-uncompressed`	`109189140.79150793` ns (`530680.3305972293`)	`112797928.43674605` ns (`1529144.2792797536`)	`0.97`

This comment was automatically generated by workflow using github-action-benchmark.

a10y · 2024-09-20T23:09:30Z

Seems like performance difference from the change is within margin of error for the benchmarks. Perhaps there are other places this can be used, like the children vector ArrayData, and correspondingly in VarBinArray (see apache/arrow-rs#6408 for recent discussion about doing this in Arrow)

robert3005 · 2024-09-20T23:11:24Z

ArrayData is the more common case. Views will be promoted to data pretty quickly if you perform operations on them

robert3005 · 2024-09-20T23:11:50Z

~~Fwiw if you want to try StructArray#project as well~~ project is a bit special in how it selects things

a10y · 2024-09-23T22:28:14Z

vortex-serde/src/messages.rs

@@ -18,7 +18,7 @@ pub enum IPCMessage<'a> {

 pub struct IPCSchema<'a>(pub &'a DType);
 pub struct IPCBatch<'a>(pub &'a Array);
-pub struct IPCArray<'a>(pub &'a Array);
+pub struct IPCArray<'a>(pub &'a Array, usize);


the second argument here is the cumulative buffer index observed before array

a10y · 2024-09-24T14:56:07Z

The change to the IPC format to change has_buffer -> buffer_index seems to have improved pretty much all of the queries.

Q3, Q4, and Q21 in the PR benchmarks run actually got slower, but after multiple runs I'm unable to replicate that locally:

aduffy@DuffyProBook ~/c/vortex (aduffy/shared-vec)> critcmp before2 after2
group                              after2                                 before2
-----                              ------                                 -------
tpch_q1/vortex-file-compressed     1.00    464.9±8.26ms        ? ?/sec    1.03   477.6±16.36ms        ? ?/sec
tpch_q10/vortex-file-compressed    1.00    405.4±3.96ms        ? ?/sec    1.08    437.0±9.49ms        ? ?/sec
tpch_q11/vortex-file-compressed    1.00    132.1±9.72ms        ? ?/sec    1.12   147.6±12.85ms        ? ?/sec
tpch_q12/vortex-file-compressed    1.01   510.0±46.01ms        ? ?/sec    1.00   504.3±17.87ms        ? ?/sec
tpch_q13/vortex-file-compressed    1.00    136.9±3.33ms        ? ?/sec    1.08    147.7±6.12ms        ? ?/sec
tpch_q14/vortex-file-compressed    1.00     75.7±4.81ms        ? ?/sec    1.21     91.2±5.38ms        ? ?/sec
tpch_q15/vortex-file-compressed    1.00    130.5±4.44ms        ? ?/sec    1.38   179.5±28.35ms        ? ?/sec
tpch_q16/vortex-file-compressed    1.00     91.9±2.16ms        ? ?/sec    1.18   108.5±10.52ms        ? ?/sec
tpch_q17/vortex-file-compressed    1.00    325.4±5.26ms        ? ?/sec    1.17   382.0±35.44ms        ? ?/sec
tpch_q18/vortex-file-compressed    1.00   606.7±23.70ms        ? ?/sec    1.07   651.8±33.28ms        ? ?/sec
tpch_q19/vortex-file-compressed    1.00   612.6±43.75ms        ? ?/sec    1.08   664.5±43.12ms        ? ?/sec
tpch_q2/vortex-file-compressed     1.00     88.8±3.26ms        ? ?/sec    1.55   137.3±30.17ms        ? ?/sec
tpch_q20/vortex-file-compressed    1.00    216.2±9.73ms        ? ?/sec    1.03    223.1±7.02ms        ? ?/sec
tpch_q21/vortex-file-compressed    1.02  1163.8±47.95ms        ? ?/sec    1.00  1144.4±30.49ms        ? ?/sec
tpch_q22/vortex-file-compressed    1.00     80.7±2.34ms        ? ?/sec    1.03     83.5±1.55ms        ? ?/sec
tpch_q3/vortex-file-compressed     1.00    367.2±3.56ms        ? ?/sec    1.08   396.0±12.09ms        ? ?/sec
tpch_q4/vortex-file-compressed     1.00    316.2±4.71ms        ? ?/sec    1.06   335.2±10.87ms        ? ?/sec
tpch_q5/vortex-file-compressed     1.00    202.0±3.93ms        ? ?/sec    1.09    220.6±7.11ms        ? ?/sec
tpch_q6/vortex-file-compressed     1.00     68.8±2.04ms        ? ?/sec    1.05     72.4±4.83ms        ? ?/sec
tpch_q7/vortex-file-compressed     1.00    483.5±7.25ms        ? ?/sec    1.15   554.1±36.47ms        ? ?/sec
tpch_q8/vortex-file-compressed     1.00    211.7±3.07ms        ? ?/sec    1.06    223.4±3.39ms        ? ?/sec
tpch_q9/vortex-file-compressed     1.00    329.2±5.57ms        ? ?/sec    1.09   359.0±19.73ms        ? ?/sec

robert3005 · 2024-09-24T15:06:52Z

Ship it. Didn’t know that we get cumulative nbuffers so often

a10y · 2024-09-24T15:51:06Z

I've updated the PR to remove SharedVec, since we were no longer slicing anything, instead opting to convert some of the Vec<Buffer> and Vec<Array> to Arc<[T]> equivalents

a10y · 2024-09-24T15:52:13Z

vortex-array/src/view.rs

        Ok(Self {
            encoding,
            dtype: dtype.clone(),
            len,
            flatbuffer: self.flatbuffer.clone(),
            flatbuffer_loc,
-            buffers: self.buffers[buffer_offset..][0..buffer_count].to_vec(),
+            buffers: self.buffers.clone(),


note that we no longer need to slice, b/c each Array fb message contains the buffer_index field, which is a global offset into the buffers list

a10y · 2024-09-24T15:54:48Z

vortex-flatbuffers/src/generated/array.rs

@@ -93,7 +93,7 @@ impl<'a> flatbuffers::Verifiable for Version {

 impl flatbuffers::SimpleToVerifyInSlice for Version {}
 pub enum ArrayOffset {}
-#[derive(Copy, Clone, PartialEq, Eq)]
+#[derive(Copy, Clone, PartialEq)]


I'm not sure why the generator stripped Eq this time, but I don't think it matters

a10y · 2024-09-24T15:55:22Z

vortex-dtype/src/dtype.rs

@@ -190,6 +190,7 @@ impl StructDType {
                }
            };

+            // TODO: do this without the extra allocations.


vortex-array/src/view.rs

a10y · 2024-09-24T17:16:28Z

Hm, things seem to have gotten worse in the most recent benchmark run, despite it mainly just being deleting the SharedVec code. Several queries got 20-40% slower.

I can't replicate this level of slowdown locally

robert3005 · 2024-09-24T17:39:10Z

As long as local run is faster I wouldn’t worry about one off runs. They're highly noisy

Implements a new SharedVec type that allows for owned, zero-copy slicing of a collection. I replaced usage of Vec<Buffer> inside of ArrayView with the SharedVec.

a10y force-pushed the aduffy/shared-vec branch from 7efdbf7 to d3c214e Compare September 20, 2024 21:06

a10y added the benchmark Run benchmarks on this branch label Sep 20, 2024

github-actions bot removed the benchmark Run benchmarks on this branch label Sep 20, 2024

github-actions bot reviewed Sep 20, 2024

View reviewed changes

a10y force-pushed the aduffy/shared-vec branch from 01318c3 to 036bd86 Compare September 23, 2024 17:13

a10y added the benchmark Run benchmarks on this branch label Sep 23, 2024

github-actions bot removed the benchmark Run benchmarks on this branch label Sep 23, 2024

a10y commented Sep 23, 2024

View reviewed changes

a10y added the benchmark Run benchmarks on this branch label Sep 24, 2024

github-actions bot removed the benchmark Run benchmarks on this branch label Sep 24, 2024

a10y changed the title ~~feat: new SharedVec for zero-copy slices~~ feat: update IPC format to hold buffer_index Sep 24, 2024

a10y marked this pull request as ready for review September 24, 2024 15:49

a10y commented Sep 24, 2024

View reviewed changes

a10y added the benchmark Run benchmarks on this branch label Sep 24, 2024

github-actions bot removed the benchmark Run benchmarks on this branch label Sep 24, 2024

robert3005 reviewed Sep 24, 2024

View reviewed changes

vortex-array/src/view.rs Outdated Show resolved Hide resolved

robert3005 approved these changes Sep 24, 2024

View reviewed changes

a10y added the benchmark Run benchmarks on this branch label Sep 24, 2024

github-actions bot removed the benchmark Run benchmarks on this branch label Sep 24, 2024

a10y added 6 commits September 24, 2024 14:59

feat: new SharedVec for zero-copy slices

4afb454

Implements a new SharedVec type that allows for owned, zero-copy slicing of a collection. I replaced usage of Vec<Buffer> inside of ArrayView with the SharedVec.

ArrayData use SharedVec to hold children

2205678

feat: ChunkedArray::chunk uses scalar_at_unchecked

dfbaba3

change IPC format to store buffer_index directly

17b4ca9

fix issue with off-by-one when top-level array has buffer

1502d77

remove SharedVec -> Arc<[Array]>

d2315ea

a10y force-pushed the aduffy/shared-vec branch from 8b218cd to d2315ea Compare September 24, 2024 18:59

remove old cumulative_nbuffers check

98f3dc4

a10y merged commit b2d4dce into develop Sep 24, 2024
5 checks passed

a10y deleted the aduffy/shared-vec branch September 24, 2024 20:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: update IPC format to hold buffer_index #903

feat: update IPC format to hold buffer_index #903

a10y commented Sep 20, 2024 •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

a10y commented Sep 20, 2024

robert3005 commented Sep 20, 2024

robert3005 commented Sep 20, 2024 •

edited

Loading

a10y Sep 23, 2024 •

edited

Loading

a10y commented Sep 24, 2024

robert3005 commented Sep 24, 2024

a10y commented Sep 24, 2024

a10y Sep 24, 2024

a10y Sep 24, 2024

a10y Sep 24, 2024

a10y commented Sep 24, 2024 •

edited

Loading

robert3005 commented Sep 24, 2024

feat: update IPC format to hold buffer_index #903

feat: update IPC format to hold buffer_index #903

Conversation

a10y commented Sep 20, 2024 • edited Loading

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

DataFusion

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

Vortex bytes_at

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

Random Access

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

Vortex Compression

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

TPC-H

a10y commented Sep 20, 2024

robert3005 commented Sep 20, 2024

robert3005 commented Sep 20, 2024 • edited Loading

a10y Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

a10y commented Sep 24, 2024

robert3005 commented Sep 24, 2024

a10y commented Sep 24, 2024

a10y Sep 24, 2024

Choose a reason for hiding this comment

a10y Sep 24, 2024

Choose a reason for hiding this comment

a10y Sep 24, 2024

Choose a reason for hiding this comment

a10y commented Sep 24, 2024 • edited Loading

robert3005 commented Sep 24, 2024

a10y commented Sep 20, 2024 •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

github-actions bot left a comment •

edited

Loading

robert3005 commented Sep 20, 2024 •

edited

Loading

a10y Sep 23, 2024 •

edited

Loading

a10y commented Sep 24, 2024 •

edited

Loading