Unable to Get ECR Creds - context deadline exceeded

[mzameer777]

What happened:
I'm trying to implement image pull from private ECR, I have installed and configured ecr-credential-provider plugin, I'm getting this error in kubelet logs, and can't figure out how to proceed further

{"ts":1727300113642.7864,"caller":"plugin/plugin.go:235","msg":"Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image XXX.dkr.ecr.us-west-2.amazonaws.com/cilium/cilium: context deadline exceeded: I0925 21:34:13.657531    6329 main.go:129] Getting creds for private image XXX.dkr.ecr.us-west-2.amazonaws.com/cilium/cilium\nW0925 21:34:13.657585    6329 main.go:65] No region found in the image reference, the default region will be used. Please refer to AWS SDK documentation for configuration purpose."}

The plugin binary is executed and it says context deadline exceeded.

below is my configuration, I'm using Talos, so this is the creds config patch

machine:
  kubelet:
    credentialProviderConfig:
      apiVersion: kubelet.config.k8s.io/v1
      kind: CredentialProviderConfig
      providers:
        - name: ecr-credential-provider
          matchImages:
            - "*.dkr.ecr.*.amazonaws.com"
          defaultCacheDuration: "12h"
          apiVersion: credentialprovider.kubelet.k8s.io/v1

Environment:

• Kubernetes version (use kubectl version):
  Client Version: v1.31.1
  Kustomize Version: v5.4.2
  Server Version: v1.30.3
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release):
  NAME="Talos"
  ID=talos
  VERSION_ID=v1.8.0
  PRETTY_NAME="Talos (v1.8.0)"
  HOME_URL="https://www.talos.dev/"
  BUG_REPORT_URL="https://github.com/siderolabs/talos/issues"
  VENDOR_NAME="Sidero Labs"
  VENDOR_URL="https://www.siderolabs.com/"

/kind bug

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cartermckinnon]

You're probably getting the timeout here:

cloud-provider-aws/cmd/ecr-credential-provider/main.go

Line 139 in d7e05d5

output, err := e.ecr.GetAuthorizationToken(&ecr.GetAuthorizationTokenInput{})

Do your nodes have network access to the ECR endpoint?

(the warning you're seeing in the output there is misleading but harmless. Fix for that in #1030)

[mzameer777]

I can confirm that the node has network connectivity to ECR VPC endpoint and it also has ECR full permissions.

What else can I look for, is there a way to debug this further in my env?

[cartermckinnon]

You verified that aws ecr get-login-password works on the node?

You can try to reproduce the cred provider failure with something like:

echo '{"kind":"CredentialProviderRequest","apiVersion":"credentialprovider.kubelet.k8s.io/v1","image":"$IMAGE"}' | ecr-credential-provider