Google Cloud Storage encoded-key problem

[fragaLY]

Hello, seems like the issue is still there.

The example of my setup doesn't works correctly:
1st of all I've created a key for gcp storage and encoded it using (macOS):
base64 -i key.json -b -

After that I've passed the encoded key into profile:
Encoded key sample:
eAiLS0tLS1CRUdJT.....SQBSFIJ!N KF!LJA=

My application.yaml:

server:
  port: 8081
  undertow:
    threads:
      io: 2
      worker: 16
    max-http-post-size: -1B
    buffer-size: 16KB
    direct-buffers: on
  shutdown: graceful
  servlet:
    session:
      timeout: 60m
      cookie:
        http-only: true
    application-display-name: api
  error:
    whitelabel:
      enabled: false

spring:
  application:
    name: api
  main:
    lazy-initialization: true
    web-application-type: servlet
    banner-mode: off
  jackson:
    time-zone: UTC
    locale: en_US
  datasource:
    driverClassName: org.postgresql.Driver
    url: ${DB_URL}
    username: ${DB_USER}
    password: ${DB_PASSWORD}
    hikari:
      connection-timeout: 1000
      pool-name: "hikari-pool"
  flyway:
    enabled: true
    locations: classpath:db/migration
    url: ${DB_FLYWAY_URL}
    user: ${DB_USER}
    password: ${DB_PASSWORD}
    connect-retries: 3
    schemas: "api"
  data:
    jpa:
      repositories:
        bootstrap-mode: deferred
  jpa:
    open-in-view: false
    hibernate:
      ddl-auto: none
    generate-ddl: false
    database-platform: org.hibernate.dialect.PostgreSQL95Dialect
    properties:
      hibernate:
        default_schema: ${DB_SCHEMA}
    show-sql: true
  mvc:
    format:
      date: iso
      date-time: iso
  lifecycle:
    timeout-per-shutdown-phase: 30s
  autoconfigure:
    exclude: org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
  servlet:
    multipart:
      enabled: true
      max-file-size: -1
      max-request-size: -1
      resolve-lazily: true
  cloud:
    gcp:
      storage:
        enabled: false

management:
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true
  endpoint:
    health:
      enabled: true
      probes:
        enabled: true
      show-components: never
      show-details: never
      group:
        readiness:
          include: readinessState, db
    metrics.enabled: true
    prometheus.enabled: true
  endpoints.web.exposure.include: "*"
  metrics.export.prometheus.enabled: true

logging.level:
  ROOT: info
  by.vk.api: info
  org.springframework: info

springdoc:
  show-actuator: true
  api-docs:
    enabled: true
    groups:
      enabled: true

---
spring:
  config:
    activate:
      on-profile: prod
  datasource:
    url: ${DB_PROD_URL}
    username: ${DB_PROD_USER}
    password: ${DB_PROD_PASSWORD}
  flyway:
    enabled: true
    url: ${DB_PROD_FLYWAY_URL}
    user: ${DB_PROD_USER}
    password: ${DB_PROD_PASSWORD}
  jpa:
    show-sql: false
  cloud:
    gcp:
      storage:
        enabled: true
        project-id: ${GCP_PROJECT_ID}
        auto-create-files: true
        credentials:
          encoded-key: eAiLS0tLS1CRUdJT.....SQBSFIJ!N KF!LJA=

If I replace encoded-key with key.json location: key.json it works the same way incorrect.

The exception:
Failed to instantiate [com.google.cloud.storage.Storage]: Factory method 'googleCloudStorage' threw exception; nested exception is java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

The configuration class that is not able to instantiate credentials:

@Profile("prod")
@Configuration
public class GoogleCloudStorageConfiguration {

 @Bean
 @ConditionalOnProperty("spring.cloud.gcp.storage.enabled")
 public GoogleCloudStorageProperties googleCloudStorageProperties() {
   return new GoogleCloudStorageProperties();
 }

 @Bean
 @ConditionalOnBean(GoogleCloudStorageProperties.class)
 public Storage googleCloudStorage(CredentialsProvider credentials, GcpProjectIdProvider project) throws IOException {
   return StorageOptions.newBuilder()
       .setCredentials(credentials.getCredentials())
       .setProjectId(project.getProjectId())
       .setRetrySettings(RetrySettings.newBuilder().setMaxAttempts(3).build())
       .build()
       .getService();
 }
}

The part of service layer I use to read blobs:

public Resource get(String name) {
    var location = forFile(properties.bucket, name);
    var blob = new GoogleStorageResource(storage, location, false);
    LOGGER.info("[BLOB] Blob with [${}] successfully found.", name);
    try(var is = blob.getInputStream()) {
      return new InputStreamResource(is);
    } catch (IOException exception) {
      throw new UnexpectedException(exception.getMessage());
    }
  }

[elefeint]

@fragaLY The custom googleCloudStorage method is expecting a generic CredentialsProvider that would be instantiated using the Core spring.cloud.gcp.credentials.encoded-key property. However, you are providing the Storage-specific spring.cloud.gcp.storage.credentials.encoded-key property that does not result in a DefaultCredentialsProvider bean being present in the application context.

You have two choices -- provide the generic spring.cloud.gcp.credentials.encoded-key property or provide a custom instance of DefaultCredentialsProvider that's based on the storage-specific property. See how the built-in autoconfiguration handles the creation of the storage object.

[BenjaminEckardt]

@elefeint Thanks for the response, switching to spring.cloud.gcp.credentials.encoded-key (not storage specific) really helped me while I was having the same problem.
If I'm not mistaken the configuration via spring.cloud.gcp.storage.credentials.encoded-key is explicitly mentioned here. Is it an error is the docs?

[elefeint]

@BenjaminEckardt No -- the documentation is correct, as long as you rely on the autoconfigured Storage bean. With built-in autoconfiguration, spring.cloud.gcp.storage.credentials.encoded-key will take priority, if present. But the usecase in this issue is when you want to configure your own Storage bean. In this case, you'd have to read the property yourself and construct the credentials provider based on it.