[HUDI-8212] Added extra config for Billing big query project id #11956
Conversation
| public static final ConfigProperty<String> BIGQUERY_SYNC_BILLING_PROJECT_ID = ConfigProperty | ||
| .key("hoodie.gcp.bigquery.sync.project_id") | ||
| .noDefaultValue() | ||
| .markAdvanced() |
| @@ -58,6 +58,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable | |||
| .markAdvanced() | |||
| .withDocumentation("Name of the target project in BigQuery"); | |||
|
|
|||
| public static final ConfigProperty<String> BIGQUERY_SYNC_BILLING_PROJECT_ID = ConfigProperty | |||
| .key("hoodie.gcp.bigquery.sync.project_id") | |||
There was a problem hiding this comment.
| .key("hoodie.gcp.bigquery.sync.project_id") | |
| .key("hoodie.gcp.bigquery.sync.billing_project_id") |
| @@ -47,6 +48,7 @@ public class TestBigQuerySyncConfig { | |||
| public void testGetConfigs() { | |||
| Properties props = new Properties(); | |||
| props.setProperty(BIGQUERY_SYNC_PROJECT_ID.key(), "fooproject"); | |||
| props.setProperty(BIGQUERY_SYNC_BILLING_PROJECT_ID.key(), "fooproject"); | |||
There was a problem hiding this comment.
Could you use a different project ID for billing in the test?
| @@ -62,6 +62,7 @@ | |||
|
|
|||
| public class TestHoodieBigQuerySyncClient { | |||
| private static final String PROJECT_ID = "test_project"; | |||
| private static final String BILLING_PROJECT_ID = "test_project"; | |||
| @@ -124,7 +128,7 @@ public void createOrUpdateTableUsingBqManifestFile(String tableName, String bqMa | |||
| QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query) | |||
| .setUseLegacySql(false) | |||
| .build(); | |||
| JobId jobId = JobId.newBuilder().setProject(projectId).setRandomJob().build(); | |||
| JobId jobId = JobId.newBuilder().setProject(billingProjectId).setRandomJob().build(); | |||
There was a problem hiding this comment.
Why is the billing project ID used here only?
There was a problem hiding this comment.
billing project id is just used for submitted the jobs. the dataset still lies in another project id.
There was a problem hiding this comment.
Got it, so it should only be used bycreateOrUpdateTableUsingBqManifestFile. Could you update the config docs to reflect this so the user knows how this new billing project ID is used?
There was a problem hiding this comment.
@yihua I will raise the separate docs PR today.
There was a problem hiding this comment.
You can directly update
.withDocumentation("Name of the billing project id in BigQuery. By default it will use the BIGQUERY_SYNC_PROJECT_ID");
When we cut release docs, the config docs is going to be automatically updated.
Change Logs
When the big query dataset project id is different than billing project id, the job has to be submitted in different project.
Impact
none
Risk level (write none, low medium or high below)
low
Documentation Update
Big Query sync Documentation update is necessary for the extra config. Will raise a PR for that too.
Contributor's checklist