[idea] Support for Rapids (GPU) with Spark Thrift in DBT

[sjabber]

Which of these best describes you?

• dbt Community member
• Partner
• dbt Labs employee
• Other

What's your idea for new content?

Currently, DBT does not support the combination of the Rapids library (GPU) with Spark Thrift, even though this setup could significantly enhance performance. This lack of support prevents the DBT + Spark (Thrift) combination from reaching its full potential in terms of processing speed and efficiency.

It would be greatly beneficial if DBT could be improved to support running dbt run with Spark Thrift when using the Rapids library (GPU). This would enable better performance, taking advantage of GPU acceleration in Spark.

When attempting to use dbt run with Spark Thrift that utilizes the Rapids library, the following error occurs:
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/RuleNotFoundExprMeta
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)

My Environment Details:

• DBT version : 1.8.3
• Spark version : 3.5.0
• Rapids : rapids-4-spark_2.12-23.12.0.jar

Where would you recommend this content live on the docs.getdbt.com?

No response