rapids to 23.06

[eordentlich]

local (fast) tests pass, need to verify cloud.
Note that dataproc image had to be bumped to get python to > 3.8 which is dropped in rapids 23.06.
rapids-4-spark plugin 23.06 update pending push to maven.

[lijinf2]

Looks good overall.

Does dataproc work with RAPIDS-23.06? Only the RAPIDS version gets upgraded, and python version remains the same (python 3.8?).

[wbo4958]

I'm wondering how you make this PR. seems there're lots of places need to be updated.

[eordentlich]

build

[eordentlich]

Verified that after patch-release updates of rapids pip packages, benchmarks run on databricks, dataproc and emr.
Dataproc required patching pyarrow in the conda environment, update to 2.1, and a minor edit to the spark-rapids install script which was added to our dataproc benchmark script directory.

[eordentlich]

build

[pxLi]

build

[pxLi]

re-triggered, seems some internal networking issue

The TLS connection was non-properly terminated

[eordentlich]

build

[pxLi]

failed

[2023-07-10T04:43:03.414Z] �[0m�[91mTraceback (most recent call last):
[2023-07-10T04:43:03.414Z]   File "/root/miniconda3/bin/mamba", line 7, in <module>
[2023-07-10T04:43:03.414Z]     from mamba.mamba import main
[2023-07-10T04:43:03.414Z]   File "/root/miniconda3/lib/python3.8/site-packages/mamba/mamba.py", line 53, in <module>
[2023-07-10T04:43:03.414Z]     import mamba.mamba_api as api
[2023-07-10T04:43:03.414Z] ImportError: libarchive.so.13: cannot open shared object file: No such file or directory
[2023-07-10T04:43:03.414Z] �[0mThe command '/bin/sh -c conda install -c conda-forge mamba &&     mamba install -y -c rapidsai -c nvidia -c conda-forge cuml=$CUML_VER python=3.9 cuda-toolkit=11.5     && mamba clean --all -f -y' returned a non-zero code: 1

we saw this often in nightly docker images build, seems some bad timing about remote mamba cache, could try later

[pxLi]

build

[eordentlich]

build

[pxLi]

build

[eordentlich]

build

[lijinf2]

It seems have to copy third-party code in spark_rapids.sh. Will be wonderful if there is a way to avoid copy (can be addressed in the future). Overall looks good.

[eordentlich]

build

[lijinf2]

Thanks for testing all of databricks, gcp dataproc and aws emr. Looks nice!