Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: support virtual-host-style links for non-AWS endpoints #38

Open
krokicki opened this issue Jul 30, 2024 · 0 comments
Open

Comments

@krokicki
Copy link

krokicki commented Jul 30, 2024

We have an S3 proxy service which emulates the S3 protocol, and serves links which work in N5 Viewer, e.g. https://s3proxy.janelia.org/shroff-public/shroff...2xbin.n5/

@axtimwalde asked me to look into supporting virtual-host-style links, since AWS has indicated that it may deprecated path-style links in the future.

Virtual-host-style links move the bucket name in the domain name. With our service, that might look like:
https://shroff-public.s3proxy.janelia.org/shroff_2022...6x6x6nm_2xbin.n5/

I implemented it in my dev environment but it doesn't work in N5 Viewer because some places in the code assume path-style links when the endpoint is not AWS:
https://github.com/saalfeldlab/n5-aws-s3/blob/master/src/main/java/org/janelia/saalfeldlab/n5/s3/AmazonS3Utils.java#L50
https://github.com/saalfeldlab/n5-aws-s3/blob/master/src/main/java/org/janelia/saalfeldlab/n5/s3/AmazonS3Utils.java#L137

A few more observations:

  1. When your URIs are not "official" AWS S3 URIs, there is no way to determine which style is being used from the URI alone. Some kind of test requests would need to be sent to determine the style. Another possibility is to enforce the <bucketname>.s3. convention that AWS uses internally, and parse the URI in a similar way to how they do it. It's not clear how compatible that would be with other services, but we could easily implement it in Janelia's S3 proxy.
  2. The AWS SDK for Java doesn't make it super intuitive to work with virtual-host-style links for non-AWS endpoints, but it does seem possible. For example, if if your endpoint is https://shroff-public.s3.janelia.org then you just need to use an empty bucket name to list the contents of the bucket: listObjectsV2("")
  3. Since non-AWS endpoints are a separate path in the code, it's not likely that they would be affected even if AWS deprecates path-style URIs for the official S3 service. Therefore, I would consider this feature a low priority from the future-proofing standpoint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant