Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support google cloud storage as das #2643

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

renlulu
Copy link
Contributor

@renlulu renlulu commented Sep 6, 2024

No description provided.

@cla-bot cla-bot bot added the s Automatically added by the CLA bot if the creator of a PR is registered as having signed the CLA. label Sep 6, 2024
Copy link
Member

@Tristan-Wilson Tristan-Wilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work so far! It just needs some additional testing and addressing my comments, which are fairly minor.

das/google_cloud_storage_service.go Outdated Show resolved Hide resolved
go.mod Show resolved Hide resolved
das/google_cloud_storage_service.go Outdated Show resolved Hide resolved
@renlulu
Copy link
Contributor Author

renlulu commented Sep 7, 2024

Proof of working:
image
image

@renlulu renlulu force-pushed the feat/das-google-cloud branch 2 times, most recently from a9aee5f to 0750428 Compare September 9, 2024 19:52
das/google_cloud_storage_service.go Outdated Show resolved Hide resolved
das/google_cloud_storage_service.go Outdated Show resolved Hide resolved
cmd/genericconf/config.go Show resolved Hide resolved
Comment on lines +68 to +69
type GoogleCloudStorageConfig struct {
AccessToken string `koanf:"access-token"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pattern for the other storage backend types is to have an Enable bool field, defaulting to false.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it looks like you've got duplicated structs that do almost the same thing. GoogleCloudStorageServiceConfig is the other one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry but the is one for unit tests only? also defaultStorageRetention not exported from das, so cannot use it here(genericconf)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was referring the s3 part, so GoogleCloudStorageConfig for ut, GoogleCloudStorageServiceConfig is for actual usage, hope i didn't get it wrongly : D

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while GoogleCloudStorageServiceConfig follow the same pattern

type GoogleCloudStorageServiceConfig struct {
	Enable       bool          `koanf:"enable"`
	AccessToken  string        `koanf:"access-token"`
	Bucket       string        `koanf:"bucket"`
	ObjectPrefix string        `koanf:"object-prefix"`
	EnableExpiry bool          `koanf:"enable-expiry"`
	MaxRetention time.Duration `koanf:"max-retention"`
}

@renlulu renlulu force-pushed the feat/das-google-cloud branch 4 times, most recently from 261a80f to 98d42fa Compare September 16, 2024 18:07
@renlulu
Copy link
Contributor Author

renlulu commented Sep 20, 2024

@Tristan-Wilson Hey can we follow up here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
s Automatically added by the CLA bot if the creator of a PR is registered as having signed the CLA.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants