Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset #730

atheriel · 2024-01-02T18:06:35Z

Attempting to use ambient credentials furnished by OIDC (e.g. on EKS or in Posit Workbench) does not work correctly without explicitly setting the default region in an environment variable.

To illustrate this, you'd expect that role assumption would work when AWS_ROLE_ARN, AWS_ROLE_SESSION_NAME, and AWS_WEB_IDENTITY_TOKEN_FILE are set:

> grep("AWS_", names(Sys.getenv()), value = TRUE, fixed = TRUE)
[1] "AWS_ROLE_ARN"                "AWS_ROLE_SESSION_NAME"       "AWS_WEB_IDENTITY_TOKEN_FILE"
> paws::sts()$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

For reference, this works fine with the AWS CLI (and every other SDK I know of):

$ env | grep AWS
AWS_ROLE_ARN=<omitted>
AWS_ROLE_SESSION_NAME=<omitted>
AWS_WEB_IDENTITY_TOKEN_FILE=<omitted>
$ aws sts get-caller-identity
{
    "UserId": "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>",
    "Account": "<omitted>",
    "Arn": "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"
}

Attempting to pass the region as part of the configuration does not work, either:

> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

To make it work you need to set AWS_DEFAULT_REGION or AWS_REGION:

> Sys.setenv("AWS_DEFAULT_REGION" = "us-west-1")
> paws::sts()$get_caller_identity()
$UserId
[1] "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>"

$Account
[1] "<omitted>"

$Arn
[1] "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"

It seems like paws should be able to skip the region requirement here, as with other SDKs and the CLI.

The text was updated successfully, but these errors were encountered:

DyfanJones · 2024-01-02T18:11:08Z

It is interesting to see this failing:

> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

are you able to do a traceback for me. Also can you include the version of paws.common you are using :)

DyfanJones · 2024-01-02T18:18:36Z

library(paws)
options(paws.log_level = 3L)
sts(config = list(region = "us-west-2"))$get_caller_identity()
#> INFO [2024-01-02 18:15:14.356]: -> POST / HTTP/1.1
#> -> Host: sts.us-west-2.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> User-Agent: paws/0.6.4 (R4.3.2; darwin20; aarch64)
#> -> Accept: application/xml
#> -> Content-Type: application/x-www-form-urlencoded; charset=utf-8
#> -> Content-Length: 43
#> -> X-Amz-Date: 20240102T181513Z
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Omitted/20240102/us-west-2/sts/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=Omitted
#> -> 
#> INFO [2024-01-02 18:15:14.356]: >> Action=GetCallerIdentity&Version=2011-06-15
#> 
#> INFO [2024-01-02 18:15:14.550]: <- HTTP/1.1 200 OK
#> INFO [2024-01-02 18:15:14.550]: <- x-amzn-RequestId: bae0b6f0-f0ba-4ec6-9afd-604667c9495a
#> INFO [2024-01-02 18:15:14.550]: <- Content-Type: text/xml
#> INFO [2024-01-02 18:15:14.551]: <- Content-Length: 408
#> INFO [2024-01-02 18:15:14.551]: <- Date: Tue, 02 Jan 2024 18:15:14 GMT
#> INFO [2024-01-02 18:15:14.551]: <- 
#> $UserId
#> [1] "Omitted"
#> 
#> $Account
#> [1] "123456789"
#> 
#> $Arn
#> [1] "arn:aws:iam::123456789:user/Omitted"

sts()$get_caller_identity()
#> INFO [2024-01-02 18:15:14.823]: -> POST / HTTP/1.1
#> -> Host: sts.eu-west-1.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> User-Agent: paws/0.6.4 (R4.3.2; darwin20; aarch64)
#> -> Accept: application/xml
#> -> Content-Type: application/x-www-form-urlencoded; charset=utf-8
#> -> Content-Length: 43
#> -> X-Amz-Date: 20240102T181514Z
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Omitted/20240102/eu-west-1/sts/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=Omitted
#> -> 
#> INFO [2024-01-02 18:15:14.824]: >> Action=GetCallerIdentity&Version=2011-06-15
#> 
#> INFO [2024-01-02 18:15:14.873]: <- HTTP/1.1 200 OK
#> INFO [2024-01-02 18:15:14.873]: <- x-amzn-RequestId: a1667bd4-0990-44af-980c-1f09d98ce54a
#> INFO [2024-01-02 18:15:14.874]: <- Content-Type: text/xml
#> INFO [2024-01-02 18:15:14.874]: <- Content-Length: 408
#> INFO [2024-01-02 18:15:14.874]: <- Date: Tue, 02 Jan 2024 18:15:14 GMT
#> INFO [2024-01-02 18:15:14.874]: <- 
#> $UserId
#> [1] "Omitted"
#> 
#> $Account
#> [1] "123456789"
#> 
#> $Arn
#> [1] "arn:aws:iam::123456789:user/Omitted"

^{Created on 2024-01-02 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.2 (2023-10-31)
#>  os       macOS Sonoma 14.0
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/London
#>  date     2024-01-02
#>  pandoc   3.1.9 @ /opt/homebrew/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package                * version date (UTC) lib source
#>  cli                      3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
#>  crayon                   1.5.2   2022-09-29 [1] CRAN (R 4.3.0)
#>  curl                     5.2.0   2023-12-08 [1] CRAN (R 4.3.1)
#>  digest                   0.6.33  2023-07-07 [1] CRAN (R 4.3.0)
#>  evaluate                 0.23    2023-11-01 [1] CRAN (R 4.3.1)
#>  fastmap                  1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs                       1.6.3   2023-07-20 [1] CRAN (R 4.3.0)
#>  glue                     1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools                0.5.7   2023-11-03 [1] CRAN (R 4.3.1)
#>  httr                     1.4.7   2023-08-15 [1] CRAN (R 4.3.0)
#>  knitr                    1.45    2023-10-30 [1] CRAN (R 4.3.1)
#>  lifecycle                1.0.4   2023-11-07 [1] RSPM (R 4.3.0)
#>  magrittr                 2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  paws                   * 0.4.0   2023-09-15 [1] CRAN (R 4.3.0)
#>  paws.common              0.6.4   2023-11-11 [1] CRAN (R 4.3.1)
#>  paws.security.identity   0.4.0   2023-09-11 [1] CRAN (R 4.3.0)
#>  purrr                    1.0.2   2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache                  0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3              1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo                     1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils                  2.12.3  2023-11-18 [1] CRAN (R 4.3.1)
#>  R6                       2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp                     1.0.11  2023-07-06 [1] CRAN (R 4.3.0)
#>  reprex                   2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang                    1.1.2   2023-11-04 [1] RSPM (R 4.3.0)
#>  rmarkdown                2.25    2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi               0.15.0  2023-07-07 [1] CRAN (R 4.3.0)
#>  sessioninfo              1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler                   1.10.2  2023-08-29 [1] CRAN (R 4.3.0)
#>  vctrs                    0.6.5   2023-12-01 [1] CRAN (R 4.3.1)
#>  withr                    2.5.2   2023-10-30 [1] CRAN (R 4.3.1)
#>  xfun                     0.41    2023-11-01 [1] CRAN (R 4.3.1)
#>  xml2                     1.3.6   2023-12-04 [1] CRAN (R 4.3.1)
#>  yaml                     2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
#> 
#>  [1] /Users/dyfanjones/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Hmmm interestingly when I include the logs in these two calls, the region is getting changed when hardcoding it.

atheriel · 2024-01-02T18:42:24Z

are you able to do a traceback for me. Also can you include the version of paws.common you are using :)

Sure!

> options(paws.log_level = 3L)
> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
INFO [2024-01-02 13:41:51.754]: Unable to locate credentials file
INFO [2024-01-02 13:41:51.754]: Unable to locate config file
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided
> traceback()
18: stop("No region provided")
17: get_region(cfgs[["credentials"]][["profile"]])
16: client_config(service_name = metadata$service_name, endpoints = metadata$endpoints, 
        cfgs = cfgs, service_id = metadata$service_id)
15: new_service(.sts$metadata, handlers, config)
14: .sts$service(config)
13: svc$assume_role_with_web_identity(RoleArn = role_arn, RoleSessionName = role_session_name, 
        WebIdentityToken = web_identity_token)
12: get_assume_role_with_web_identity_creds(role_arn = get_role_arn(), 
        role_session_name = get_role_session_name(), web_identity_token = get_web_identity_token())
11: get_container_credentials_eks()
10: f()
9: call_with_args(provider, credentials)
8: get_credentials(signer$credentials)
7: sign_with_body(v4, request$http_request, request$body, name, 
       region, request$expire_time, request$expire_time > 0, signing_time)
6: sign_sdk_request_with_curr_time(request)
5: handler$fn(request)
4: run(request, sign)
3: sign(request)
2: send_request(request)
1: paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
> packageVersion("paws.common")
[1] ‘0.6.4’

DyfanJones · 2024-01-02T21:18:20Z

Thanks for the traceback, I will have a look to see if I can replicate this issue :)

DyfanJones · 2024-01-02T21:20:29Z

I think I know what is going on, but will have a deeper look at it tomorrow. I believe I can get this in 0.7.0 paws.cmmon. @atheriel are you available for testing if possible?

atheriel · 2024-01-02T21:26:44Z

@DyfanJones Absolutely, I have a few environments that look like this for testing.

DyfanJones · 2024-01-03T14:19:45Z

@atheriel I believe I have a solution, please try out:

remotes::install_github("dyfanjones/paws/paws.common", ref = "aws-global")

atheriel · 2024-01-03T17:41:47Z

STS seems to be working now!

> Sys.unsetenv("AWS_DEFAULT_REGION")
> paws::sts()$get_caller_identity()
$UserId
[1] "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>"

$Account
[1] "<omitted>"

$Arn
[1] "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"

However, regional services don't seem to work:

> paws::s3()$list_buckets()
Error in resolver_endpoint(service_name, region, endpoints, sts_regional_endpoint) : 
  No region provided and no global region found.
> traceback()
6: stop("No region provided and no global region found.")
5: resolver_endpoint(service_name, region, endpoints, sts_regional_endpoint)
4: client_config(service_name = metadata$service_name, endpoints = metadata$endpoints, 
       cfgs = cfgs, service_id = metadata$service_id)
3: new_service(.s3$metadata, handlers, config)
2: .s3$service(config)
1: paws::s3()$list_buckets()

Unless you provide region, which also works now:

> paws::s3(config = list(region = "us-west-2"))$list_buckets()
<omitted>

In contrast, aws s3 ls seems to work fine without a region. Does it use some kind of default?

DyfanJones · 2024-01-03T17:50:00Z

not sure with aws cli. however i do know aws s3 doesn't have a global region. hence it needing a region

DyfanJones · 2024-01-03T17:52:26Z

Does it work in boto3? If so I might be missing something

atheriel · 2024-01-03T18:17:10Z

Heh, it looks like this is actually a quirk of the CLI: for S3 specifically it lists buckets in all regions.

So I think paws is working correctly now. 😄

DyfanJones · 2024-01-03T18:23:26Z

that is good to hear :) when cran allows submissions after the xmas break, i will release paws.common 0.7.0 :)

DyfanJones · 2024-01-03T18:34:31Z

Oh hang on it looks like s3 service defaults to us-east-1 when region isn't set within botocore

https://github.com/boto/botocore/blob/046058dbec07e3292a88fa05e1a997573ce48709/botocore/regions.py#L189

DyfanJones · 2024-01-03T18:49:17Z

@atheriel please have a go. This should fix the s3 issue as well.

remotes::install_github("paws-r/paws/paws.common")

atheriel · 2024-01-03T19:17:36Z

🎉

> Sys.unsetenv("AWS_DEFAULT_REGION")
> paws::s3()$list_buckets()$Buckets |> length()
[1] 52

DyfanJones · 2024-01-09T10:41:04Z

Closing ticket as paws.common 0.7.0 has been released to the cran

DyfanJones added the bug 🐞 Something isn't working label Jan 2, 2024

DyfanJones mentioned this issue Jan 3, 2024

fix aws-global default region #733

Merged

DyfanJones closed this as completed Jan 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset #730

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset #730

atheriel commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

atheriel commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

atheriel commented Jan 2, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 9, 2024

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when AWS_DEFAULT_REGION is unset #730

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when AWS_DEFAULT_REGION is unset #730

Comments

atheriel commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

atheriel commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

DyfanJones commented Jan 2, 2024

atheriel commented Jan 2, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

DyfanJones commented Jan 3, 2024

atheriel commented Jan 3, 2024

DyfanJones commented Jan 9, 2024

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset #730

Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset #730