failed to upload state: operation error S3: PutObject

[johncosta]

Bug Report

Describe the bug

Given terraform version 1.6.4, when saving state using Spaces as the backend, I get a failed to upload state error.

This also occurs when using terraform version 1.5.7 and the DO provider > 2.28.0.

Using terraform 1.5.7 and provider 2.28.0 works as expected.

Affected Resource(s)

Spaces backend

Expected Behavior

Backend state is saved to Spaces as expected.

Actual Behavior

│ Error: Failed to save state
│
│ Error saving state: failed to upload state: operation error S3: PutObject,
│ https response error StatusCode: 400, RequestID:
│ tx00000afbdb0305e0080f9-00656d9299-50742-nyc3d, HostID:
│ 50742-nyc3d-nyc3-zg04, api error XAmzContentSHA256Mismatch: UnknownError

Steps to Reproduce

1. terraform apply

Terraform Configuration Files

terraform {
    backend "s3" {
      key      = "terraform-digitialocean-kubernetes/terraform.tfstate"
      bucket   = "<bucket name redacted>"
      region   = "nyc3"

      access_key = "<access key redacted>"
      secret_key = "<access key secret redacted>"
      endpoints = { s3 = "https://nyc3.digitaloceanspaces.com" }

      encrypt                     = true
      skip_region_validation      = true
      skip_credentials_validation = true
      skip_metadata_api_check     = true
      skip_requesting_account_id  = true
  }
}

Terraform version

% terraform -v
Terraform v1.6.4
on darwin_amd64
+ provider registry.terraform.io/digitalocean/digitalocean v2.28.0
+ provider registry.terraform.io/hashicorp/local v2.4.0

Your version of Terraform is out of date! The latest version
is 1.6.5. You can update by downloading from https://www.terraform.io/downloads.html

Debug Output

2023-12-04T04:14:46.686-0500 [INFO]  backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=cc212b98-944a-3803-1b80-301129c525e6 tf_backend.s3.bucket=<bucket name redacted> tf_backend.s3.path=terraform-digitialocean-kubernetes/terraform.tfstate

2023-12-04T04:14:46.689-0500 [DEBUG] backend-s3: HTTP Request Sent: aws.region=nyc3 aws.s3.bucket=<bucket name redacted> aws.s3.key=terraform-digitialocean-kubernetes/terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=cc212b98-944a-3803-1b80-301129c525e6 tf_backend.s3.bucket=<bucket name redacted> tf_backend.s3.path=terraform-digitialocean-kubernetes/terraform.tfstate http.request.header.content_encoding=aws-chunked http.request.header.x_amz_date=20231204T091446Z http.request.body="[Redacted: 943 bytes, Type: application/json]" http.method=PUT http.request.header.amz_sdk_invocation_id=1e6b98d6-7333-4884-9528-86bcbd2f66a8 http.request.header.x_amz_server_side_encryption=AES256 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=DO00YX28LWHTXW2YYTL9/20231204/nyc3/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-server-side-encryption;x-amz-trailer, Signature=*****" http.request.header.accept_encoding=identity http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER http.request.header.content_type=application/json http.url=https://<bucket name redacted>.nyc3.digitaloceanspaces.com/terraform-digitialocean-kubernetes/terraform.tfstate?x-id=PutObject http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.4 (+https://www.terraform.io) aws-sdk-go-v2/1.22.2 os/macos lang/go#1.21.3 md/GOOS#darwin md/GOARCH#amd64 api/s3#1.42.1 ft/s3-transfer" http.request.header.x_amz_trailer=x-amz-checksum-sha256 http.request.header.x_amz_decoded_content_length=863 http.request.header.amz_sdk_request="attempt=1; max=5" net.peer.name=<bucket name redacted>.nyc3.digitaloceanspaces.com http.request_content_length=943 http.request.header.x_amz_sdk_checksum_algorithm=SHA256
2023-12-04T04:14:46.870-0500 [DEBUG] backend-s3: HTTP Response Received: aws.region=nyc3 aws.s3.bucket=<bucket name redacted> aws.s3.key=terraform-digitialocean-kubernetes/terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=cc212b98-944a-3803-1b80-301129c525e6 tf_backend.s3.bucket=<bucket name redacted> tf_backend.s3.path=terraform-digitialocean-kubernetes/terraform.tfstate http.duration=181 http.status_code=400 http.response_content_length=267 http.response.header.x_amz_request_id=tx00000b473554fae011dc3-00656d9886-50742-nyc3d http.response.header.accept_ranges=bytes
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?><Error><Code>XAmzContentSHA256Mismatch</Code><Message></Message><BucketName><bucket name redacted></BucketName><RequestId>tx00000b473554fae011dc3-00656d9886-50742-nyc3d</RequestId><HostId>50742-nyc3d-nyc3-zg04</HostId></Error>
   http.response.header.content_type=application/xml http.response.header.date="Mon, 04 Dec 2023 09:14:46 GMT" http.response.header.cache_control=max-age=60 http.response.header.strict_transport_security="max-age=15552000; includeSubDomains; preload" http.response.header.x_envoy_upstream_healthchecked_cluster=""
2023-12-04T04:14:46.873-0500 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 400, RequestID: tx00000b473554fae011dc3-00656d9886-50742-nyc3d, HostID: 50742-nyc3d-nyc3-zg04, api error XAmzContentSHA256Mismatch: UnknownError: aws.region=nyc3 aws.s3.bucket=<bucket name redacted> aws.s3.key=terraform-digitialocean-kubernetes/terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=cc212b98-944a-3803-1b80-301129c525e6 tf_backend.s3.bucket=<bucket name redacted> tf_backend.s3.path=terraform-digitialocean-kubernetes/terraform.tfstate

Panic Output
n/a

Additional context

n/a

Important Factoids
n/a

References
N/a

[andrewsomething]

The code for backend state management is entirely in the upstream Terraform code base, not the DigitalOcean provider. Unfortunately HashiCorp introduced changes that broke existing configurations in their v1.6.0 release for users of "S3-like" object storage.

See: hashicorp/terraform#33983 and hashicorp/terraform#34086

I've been able to continue using Spaces as a Terraform backend by setting:

    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    skip_s3_checksum            = true

[johncosta]

Thank you for the reply and insights.

Using the settings provided has worked for me.

I'll go ahead and close this as it's unrelated to the provider.