Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--credential_helper: overly verbose error messages when a credential is missing #23473

Open
jayconrod opened this issue Aug 29, 2024 · 4 comments · May be fixed by #23656
Open

--credential_helper: overly verbose error messages when a credential is missing #23473

jayconrod opened this issue Aug 29, 2024 · 4 comments · May be fixed by #23656
Assignees
@coeuvre
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@jayconrod
Copy link
Contributor

Description of the bug:

When authenticating to a remote cluster with the --credential_helper flag, if the credential is expired or unavailable, Bazel's error message is very long and confusing:

$ bazel build --config=glass --config=remote_debian12_x64 //base/go/fileutil
INFO: Writing tracer profile to '/private/var/tmp/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/command.profile.gz'
INFO: Invocation ID: 0af6815a-c18b-4c8b-8d9f-0df2943dd684
INFO: Streaming build results to: https://glass.cluster.engflow.com/invocations/default/0af6815a-c18b-4c8b-8d9f-0df2943dd684
WARNING: Build options --cpu, --extra_execution_platforms, --extra_toolchains, and 3 more have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //base/go/fileutil:fileutil (6 packages loaded, 16582 targets configured).
ERROR: Failed to query remote execution capabilities: UNAVAILABLE: Credentials failed to obtain metadata
io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata
        at io.grpc.Status.asRuntimeException(Status.java:535)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperException: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com
Please refresh credentials by running:

    engflow_auth login glass.cluster.engflow.com


    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:122)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:107)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:79)
    at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
    at com.google.auth.Credentials$1.run(Credentials.java:98)
    ... 3 more



ERROR: /private/var/tmp/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/rules_gogo_sdk~engflow__download_0_linux_amd64/BUILD.bazel:62:15: GoToolchainBinaryBuild external/rules_gogo_sdk~engflow__download_0_linux_amd64/builder [for tool] failed: (Exit 34): UNAVAILABLE: Credentials failed to obtain metadata: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com



java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at com.google.devtools.build.lib.remote.GoogleChannelConnectionFactory$1.onFailure(GoogleChannelConnectionFactory.java:165)

at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1120)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:852)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)

at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)

at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)

at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)

at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)

at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)

at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at io.grpc.Status.asRuntimeException(Status.java:535)

... 10 more

Caused by: com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperException: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com


    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:122)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:107)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:79)
    at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
    at com.google.auth.Credentials$1.run(Credentials.java:98)
    ... 3 more



Target //base/go/fileutil:fileutil failed to build

INFO: Elapsed time: 13.222s, Critical Path: 10.41s

INFO: 2 processes: 2 internal.

ERROR: Build did NOT complete successfully

WARNING: Uploading BEP referenced local file: UNAVAILABLE: Credentials failed to obtain metadata: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com



java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at com.google.devtools.build.lib.remote.GoogleChannelConnectionFactory$1.onFailure(GoogleChannelConnectionFactory.java:165)

at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1120)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:852)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)

at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)

at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)

at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)

at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)

at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)

at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at io.grpc.Status.asRuntimeException(Status.java:535)

... 10 more

Caused by: com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperException: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com


    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:122)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:107)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:79)
    at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
    at com.google.auth.Credentials$1.run(Credentials.java:98)
    ... 3 more



WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/execution_graph_dump.proto.zst hash: "63936441052eb3ab025fe76ac644a68a46a150211b22016375022e22deb14bf1"

size_bytes: 462

: UNAVAILABLE: Credentials failed to obtain metadata: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com



java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at com.google.devtools.build.lib.remote.GoogleChannelConnectionFactory$1.onFailure(GoogleChannelConnectionFactory.java:165)

at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1120)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:852)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)

at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)

at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)

at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)

at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)

at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)

at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at io.grpc.Status.asRuntimeException(Status.java:535)

... 10 more

Caused by: com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperException: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com


    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:122)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:107)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:79)
    at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
    at com.google.auth.Credentials$1.run(Credentials.java:98)
    ... 3 more



WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/command.profile.gz hash: "d3ab628eb7e27fd3281575ca8d4f7b87b3600ac798554e0aee119f364f226ee3"

size_bytes: 7857

: UNAVAILABLE: Credentials failed to obtain metadata: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com



java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at com.google.devtools.build.lib.remote.GoogleChannelConnectionFactory$1.onFailure(GoogleChannelConnectionFactory.java:165)

at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1120)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:852)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)

at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)

at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:106)

at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)

at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)

at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)

at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)

at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)

at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)

at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)

at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)

at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)

at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

at io.grpc.Status.asRuntimeException(Status.java:535)

... 10 more

Caused by: com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperException: Failed to get credentials for 'https://glass.cluster.engflow.com/build.bazel.remote.execution.v2.Capabilities' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com


    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:122)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:107)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:79)
    at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
    at com.google.auth.Credentials$1.run(Credentials.java:98)
    ... 3 more



ERROR: The Build Event Protocol upload failed: All 4 retry attempts failed. UNAVAILABLE: Failed to get credentials for 'https://glass.cluster.engflow.com/google.devtools.build.v1.PublishBuildEvent' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com



UNAVAILABLE: Failed to get credentials for 'https://glass.cluster.engflow.com/google.devtools.build.v1.PublishBuildEvent' from helper '/Users/jay/bin/engflow_auth': process exited with code 7. stderr: Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com

Please refresh credentials by running:


    engflow_auth login glass.cluster.engflow.com

In this case, the credential helper exits with a non-zero exit code and prints this message on stderr:

Error: Missing/invalid/expired credentials for cluster: glass.cluster.engflow.com
Please refresh credentials by running:

        engflow_auth login glass.cluster.engflow.com

It looks like Bazel is attempting multiple gRPC calls, both to Capabilities and to PublishBuildEvent.

Bazel also exits with code 34, a remote error, which doesn't seem correct.

I think ideally Bazel should invoke the credential helper early when starting up, and if that fails, exit with a short message echoing the stderr output from the credential helper. Bazel shouldn't attempt to call gRPC services if it failed to obtain a credential.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

  1. Install any credential helper. We're testing this with engflow_auth. But a script that exits non-zero and prints anything on stderr would work.

  2. Build any target with:

    --credential_helper=example.com=engflow_auth
--remote_executor=example.com
--bes_backend=example.com

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

release 8.0.0-pre.20240730.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

n/a

What's the output of git remote get-url origin; git rev-parse HEAD ?

n/a

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

n/a

Have you found anything relevant by searching the web?

n/a

Any other information, logs, or outputs that you want to share?

No response
@github-actions github-actions bot added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Aug 29, 2024
@meisterT
Copy link
Member

meisterT commented Sep 2, 2024

cc @tjgq

@coeuvre
Copy link
Member

coeuvre commented Sep 3, 2024

I think ideally Bazel should invoke the credential helper early when starting up, and if that fails, exit with a short message echoing the stderr output from the credential helper. Bazel shouldn't attempt to call gRPC services if it failed to obtain a credential.

It used to be that but the tradeoff is it adds hundreds of milliseconds delay (and of course, depending on how fast is the credential helper) to each invocation. We wanted incremental builds to be fast so we moved the initialization to the background.

@jayconrod
Copy link
Contributor Author

I understand not wanting to delay the invocation by invoking the credential helper synchronously. This is a pretty confusing user experience though.

How about invoking the credential helper in the background (perhaps for the GetCapabilities call), then if it fails, only printing that error and exiting? I'm mainly just concerned about presentation of the output.

@coeuvre
Copy link
Member

coeuvre commented Sep 4, 2024

It makes sense. Let me find a way to do it.

@joeleba joeleba added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Sep 10, 2024
@coeuvre coeuvre linked a pull request Sep 18, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coeuvre coeuvre

Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Correctly propagate errors and abort the build for certain remote errors. coeuvre/bazel
7 participants
@jayconrod @meisterT @coeuvre @joeleba @sgowroji @iancha1992 @satyanandak