[GLUTEN-7031][CORE] Initialize new module structure gluten-core / gluten-substrait

[zhztheplayer]

#7031

Part of the dependency graph after the change will become

gluten-core
  \- gluten-substrait
        |- backends-clickhouse
        \- gluten-data
             \- backends-velox

This belongs to the objective supporting mixed backend execution in Gluten.

The PR only initializes the new module structure. Will move more code in subsequent PRs.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

#7031

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

cc @zzcclp @baibaichen

This removes blockers towards multi-backend support of Gluten

[github-actions]

Run Gluten Clickhouse CI

[baibaichen]

@zhztheplayer, I would prefer the follow directory structure

gluten-core
  \- gluten-substrait
        |- backends-clickhouse
        \- backends-velox
             \- velox-data 

[lwz9103]

@zhztheplayer
Gluten ClickHouse CI is failed due to
cpp-ch/local-engine/proto/substrait is linked to ../../../gluten-core/src/main/resources/substrait/proto/substrait
related pr :6c92151

Can you help me to fix it.

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

hi @baibaichen, the final goal is to implement this dep graph

gluten-core
  |- gluten-substrait
  |      \- backends-clickhouse
  \- gluten-arrow

Then backends-velox could inherit from both gluten-substrait and gluten-arrow. gluten-data may be factored out to gluten-arrow which will no longer rely on gluten-substrait.

[lwz9103]

@zhztheplayer
The good news is that the gluten ch native code has compiled successfully after modifications. However, the Java code is currently failing to compile due to changes in SoftAffinity. When initializing SoftAffinity in MixedAffinitySuite, several modified parameters should be added. Can you help me resolve this issue?

You can also use the following command locally to test whether the Java code for Gluten CH can compile successfully.
mvn clean install -DskipTests -Pbackends-clickhouse -Pspark-3.3 -Pspark-ut

[zhztheplayer]

@lwz9103 Thank you for keeping following. Let me update the PR.

[github-actions]

Run Gluten Clickhouse CI

[baibaichen]

hi @baibaichen, the final goal is to implement this dep graph

gluten-core
  |- gluten-substrait
  |      \- backends-clickhouse
  \- gluten-arrow

Then backends-velox could inherit from both gluten-substrait and gluten-arrow. gluten-data may be factored out to gluten-arrow which will no longer rely on gluten-substrait.

Which module depends on gluten-arrow? Does it only used in backends-velox?

oh, I see the idea. Is there any more new backends which depends on gluten-arrow?

[PHILO-HE]

Thanks!