[EKS] Planning fails when assuming an IAM role with "getting credentials: exec: executable aws failed with exit code 254"

[soulshake]

My plans fail with a bunch of lines like:

Error: Get "https://4BEF2107D872F8E1B7749B06522FF05D.yl4.us-east-2.eks.amazonaws.com/api?timeout=32s": getting credentials: exec: executable aws failed with exit code 254

When I inspect the debug log I can see that the get-token command in the kubeconfig produced by the module is causing my role to try to assume itself, which can be reproduced by running:

$ aws eks get-token --cluster-name foo-apps-us-east-2 --role arn:aws:iam::123456789:role/foo-ci-pr
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::123456789:assumed-role/foo-ci-pr/my-session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789:role/foo-ci-pr

$ echo $?
254

Debug log

2021-05-02T08:45:04.594Z [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/template\"]" produced an unexpected new value for module.eks_foo.module.cluster.module.cluster_services.data.template_file.kubeconfig.
      - .rendered: was cty.StringVal("apiVersion: v1\nclusters:\n- cluster:\n    server: https://ABC123.yl4.us-east-2.eks.amazonaws.com\n    certificate-authority-data: bla=\n  name: foo-apps-us-east-2\ncontexts:\n- context:\n    cluster: foo-apps-us-east-2\n    user: foo-apps-us-east-2\n  name: foo-apps-us-east-2\ncurrent-context: foo-apps-us-east-2\nkind: Config\npreferences: {}\nusers:\n- name: foo-apps-us-east-2\n  user:\n    exec:\n      apiVersion: client.authentication.k8s.io/v1alpha1\n      command: aws\n      args:\n        - \"eks\"\n        - \"get-token\"\n        - \"--cluster-name\"\n        - \"foo-apps-us-east-2\"\n        \n        - \"--role\"\n        - \"arn:aws:iam::123456789:role/infra-automation\"\n        \n"), but now cty.StringVal("apiVersion: v1\nclusters:\n- cluster:\n    server: https://ABC123.yl4.us-east-2.eks.amazonaws.com\n    certificate-authority-data: blah=\n  name: foo-apps-us-east-2\ncontexts:\n- context:\n    cluster: foo-apps-us-east-2\n    user: foo-apps-us-east-2\n  name: foo-apps-us-east-2\ncurrent-context: foo-apps-us-east-2\nkind: Config\npreferences: {}\nusers:\n- name: foo-apps-us-east-2\n  user:\n    exec:\n      apiVersion: client.authentication.k8s.io/v1alpha1\n      command: aws\n      args:\n        - \"eks\"\n        - \"get-token\"\n        - \"--cluster-name\"\n        - \"foo-apps-us-east-2\"\n        \n        - \"--role\"\n        - \"arn:aws:iam::123456789:role/infra-ci-pr\"\n        \n")
      - .id: was cty.StringVal("abc123"), but now cty.StringVal("def456")

The generated config itself:

$ terraform state show module.eks_foo.module.cluster.module.cluster_services.data.template_file.kubeconfig
# module.eks_foo.module.cluster.module.cluster_services.data.template_file.kubeconfig:
data "template_file" "kubeconfig" {
    # ...
    rendered = <<-EOT
        apiVersion: v1
        clusters:
        - cluster:
          # ...
          name: foo-apps-us-east-2
        contexts:
        - context:
            cluster: foo-apps-us-east-2
            user: foo-apps-us-east-2
          name: foo-apps-us-east-2
        current-context: foo-apps-us-east-2
        kind: Config
        preferences: {}
        users:
        - name: foo-apps-us-east-2
          user:
            exec:
              apiVersion: client.authentication.k8s.io/v1alpha1
              command: aws
              args:
                - "eks"
                - "get-token"
                - "--cluster-name"
                - "foo-apps-us-east-2"
                
                - "--role"
                - "arn:aws:iam::123456789:role/foo-ci-pr"

Provider config

This is running on an ec2 instance with the role foo-ci-pr, so is obtaining credentials via the Ec2InstanceMetadata endpoint.

$ aws sts get-caller-identity 
{
    "UserId": "AROABLABLA:i-0123eeaae",
    "Account": "123456789",
    "Arn": "arn:aws:sts::123456789:assumed-role/foo-ci-pr/i-0123eeaae"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.1.0"
    }
    # ...
  }
}

provider "aws" {
  region              = "us-east-2"
  allowed_account_ids = [123456789]
}

provider "kubernetes" {
  alias                  = "foo_cluster"
  host                   = data.aws_eks_cluster.foo.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.foo.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.foo.token
}

data "aws_eks_cluster" "foo" {
  name = local.k8s_cluster_name
}

data "aws_eks_cluster_auth" "foo" {
  name = data.aws_eks_cluster.foo.name
}

Workarounds

Planning works if grant the IAM role permission to assume itself.

It also works if I manually remove this block from ./.terraform/modules/eks_foo/aws/_modules/eks/templates/kubeconfig.tpl:

          %{ if caller_id_arn_type == "role" }
          - "--role"
          - "${caller_id_arn}"
          %{ endif }