Support for GDAL related assets #218

ahuarte47 · 2022-09-21T09:26:22Z

This PR implements #217

It provides support for loading assets from GDAL Virtual File Systems keys.

Samweli · 2022-09-23T07:52:40Z

src/qgis_stac/gui/assets_dialog.py

@@ -412,6 +412,18 @@ def sign_asset_href(self, asset_href):
            signed_href = pc.sign(asset_href)
            return signed_href

+        if asset_href.startswith("s3://"):


@ahuarte47 can you put the code for with the new changes into its own function the sign_asset_href method is for signing planetary computer SAS based items hrefs with a SAS token.

Is not the generation of signed urls the purpose of sign_asset_href? IMHO this function can manage all cases without modifying the current behavior or the existence of new future sign methods. Do you agree? Just chaning the description of the method?

Lets make another function for the new changes.

@Samweli can't really understand. The plugin and the sign method are not named as microsoft or planetarycomputer dependent => because the plugin is a Common, have to be and remain general.

sign_asset_href is a enough general name that can refer to the signing action that is common to any cloud resource not only that from microsoft services.

In resume, what is the rationale to move a sign feature for only GC or AWS to a different method respect signing of microsoft cloud?

@ahuarte47 @luipir I prefer to have another function for the new changes, at some point I ll also rename the current sign_asset_href to a more specific PC related name.

Samweli

@ahuarte47 I tried your approach with a couple of STAC catalogs that use AWS S3 service for item assets hrefs but couldn't make them work. What catalogs are you using to test this functionality?

Samweli · 2022-09-23T07:55:08Z

src/qgis_stac/gui/assets_dialog.py

@@ -412,6 +412,18 @@ def sign_asset_href(self, asset_href):
            signed_href = pc.sign(asset_href)
            return signed_href

+        if asset_href.startswith("s3://"):
+            signed_href = gdal.GetSignedURL(f"/vsis3/{asset_href[5:]}")


Couldn't find documentation for the GetSignedURL https://gdal.org/api/python/osgeo.gdal.html?highlight=getsigned#osgeo.gdal.GetSignedURL, can you explain what it tries to achieve?

gdal.GetSignedURL tranforms a GDAL virtual file system path to one https signed url. Depending on the virtual file system type, GDAL reads its specific authentication settings from environment variables. This is an example:
https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsioss.py#L185

"A signed URL is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource" from https://cloud.google.com/storage/docs/access-control/signed-urls

For example, GDAL reads several environment variables (AWS_NO_SIGN_REQUEST, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN...) to create the signed url. It is very useful custom way to auhenticate, apart from using the UI of authentication that your plugin provides.

@Samweli you can use the Collection Digital Earth Africa - s2l2a in the predefined Catalog. Before, You have to define this environment variable AWS_NO_SIGN_REQUEST=YES. Using gdal 3.5.1 it works.

So these new changes will not work without user having to define the environment variable AWS_NO_SIGN_REQUEST?

Yes. For other use cases, credentials will be configured defining AWS_PROFILE or AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY. It depends on how the STAC Catalog is deployed in the cloud-provider.

…signed url.

ahuarte47 · 2022-09-23T09:44:49Z

Hi @Samweli I have uploaded more changes. I have noticed that original asset.href attribute is changed when the plugin wants to load the asset. if the authentication expires in a short time next invocations to load the asset again will fail.

ahuarte47 · 2022-09-26T12:26:09Z

Hi @Samweli do you need more info about last changes? Can I help with anything to get this PR merged? Thanks!!

luipir · 2022-10-05T15:00:49Z

Any feedback @Samweli how to proceed ?

Samweli · 2022-10-05T15:10:38Z

Any feedback @Samweli how to proceed ?

@luipir I ll take a look at the new changes and get back to you guys later, Apologies for a slow reply, We currently don't have enough time to work on support and maintenance.

Samweli · 2022-10-12T09:23:49Z

Any feedback @Samweli how to proceed ?

@luipir @ahuarte47 I have added new comments on the review suggestions, In general I like these new changes to be implemented in a separate function, as I have plans to update the sign_asset_href in the future.

luipir · 2022-10-12T09:29:33Z

@Samweli reasonable your proposed changes if you are planning to move the code to be more agnostic an not only PC oriented :)

Samweli · 2022-10-12T09:35:25Z

@Samweli reasonable your proposed changes if you are planning to move the code to be more agnostic an not only PC oriented :)

yes @luipir in the future we ll have a sign_asset_href that will support all the url sign based operations.

luipir · 2022-11-09T21:29:10Z

please @Samweli give us a feedback regarding latest modification done by @ahuarte47, thank you

remicres · 2023-03-03T11:28:02Z

Just gave a try with private S3 endpoint: it is working great.

The only cumbersome thing is to define the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_S3_ENDPOINT environment variables

remicres · 2023-03-03T11:40:56Z

Just saw that GDAL GetSignedURL() returns an URL valid only for 1 hour (however that is not related to this PR)

slesaad · 2023-09-15T14:51:44Z

Any updates on when this PR will be merged?

Support for GDAL related assets

0d1ff28

Samweli closed this Sep 23, 2022

Samweli reopened this Sep 23, 2022

Samweli reviewed Sep 23, 2022

View reviewed changes

ahuarte47 added 2 commits September 23, 2022 10:59

Do sign_asset_href more generic

8cd9f32

Do not override asset.href, otherwise new invocations will use first …

e9862df

…signed url.

remicres mentioned this pull request Oct 28, 2022

Support custom URLs signing #223

Open

drnextgis mentioned this pull request Sep 14, 2023

Support adding assets with s3:// protocol #242

Open

yuvipanda mentioned this pull request Mar 29, 2024

Figure out plan for maintenance of QGIS STAC browser plugin NASA-IMPACT/veda-jupyterhub#12

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for GDAL related assets #218

Support for GDAL related assets #218

ahuarte47 commented Sep 21, 2022

Samweli Sep 23, 2022

ahuarte47 Sep 23, 2022 •

edited

Loading

Samweli Sep 23, 2022

luipir Sep 23, 2022

Samweli Oct 12, 2022

Samweli left a comment

Samweli Sep 23, 2022

ahuarte47 Sep 23, 2022

luipir Sep 23, 2022

ahuarte47 Sep 23, 2022

ahuarte47 Sep 23, 2022 •

edited

Loading

Samweli Oct 12, 2022

ahuarte47 Oct 12, 2022 •

edited

Loading

ahuarte47 commented Sep 23, 2022 •

edited

Loading

ahuarte47 commented Sep 26, 2022

luipir commented Oct 5, 2022

Samweli commented Oct 5, 2022

Samweli commented Oct 12, 2022 •

edited

Loading

luipir commented Oct 12, 2022

Samweli commented Oct 12, 2022

luipir commented Nov 9, 2022

remicres commented Mar 3, 2023

remicres commented Mar 3, 2023 •

edited

Loading

slesaad commented Sep 15, 2023

Support for GDAL related assets #218

Are you sure you want to change the base?

Support for GDAL related assets #218

Conversation

ahuarte47 commented Sep 21, 2022

Choose a reason for hiding this comment

ahuarte47 Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Samweli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahuarte47 Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahuarte47 Oct 12, 2022 • edited Loading

Choose a reason for hiding this comment

ahuarte47 commented Sep 23, 2022 • edited Loading

ahuarte47 commented Sep 26, 2022

luipir commented Oct 5, 2022

Samweli commented Oct 5, 2022

Samweli commented Oct 12, 2022 • edited Loading

luipir commented Oct 12, 2022

Samweli commented Oct 12, 2022

luipir commented Nov 9, 2022

remicres commented Mar 3, 2023

remicres commented Mar 3, 2023 • edited Loading

slesaad commented Sep 15, 2023

ahuarte47 Sep 23, 2022 •

edited

Loading

ahuarte47 Sep 23, 2022 •

edited

Loading

ahuarte47 Oct 12, 2022 •

edited

Loading

ahuarte47 commented Sep 23, 2022 •

edited

Loading

Samweli commented Oct 12, 2022 •

edited

Loading

remicres commented Mar 3, 2023 •

edited

Loading