{"record":{"id":"97df555c594cbdb6","repo":"hashicorp/terraform","slug":"failed-to-upload-state-w","errorCode":null,"errorMessage":"failed to upload state: %w","messagePattern":"failed to upload state: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/backend/remote-state/s3/client.go","lineNumber":240,"sourceCode":"\t\t\tinput.SSECustomerAlgorithm = aws.String(string(s3EncryptionAlgorithm))\n\t\t\tinput.SSECustomerKeyMD5 = aws.String(c.getSSECustomerKeyMD5())\n\t\t} else {\n\t\t\tinput.ServerSideEncryption = s3EncryptionAlgorithm\n\t\t}\n\t}\n\n\tif c.acl != \"\" {\n\t\tinput.ACL = s3types.ObjectCannedACL(c.acl)\n\t}\n\n\tlog.Info(\"Uploading remote state\")\n\n\tuploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {\n\t\tu.ClientOptions = optFns\n\t})\n\t_, err := uploader.Upload(ctx, input)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to upload state: %w\", err)\n\t}\n\n\tif err := c.putMD5(ctx, sum[:]); err != nil {\n\t\t// if this errors out, we unfortunately have to error out altogether,\n\t\t// since the next Get will inevitably fail.\n\t\treturn fmt.Errorf(\"failed to store state MD5: %w\", err)\n\t}\n\n\treturn nil\n}\n\nfunc (c *RemoteClient) Delete() tfdiags.Diagnostics {\n\tvar diags tfdiags.Diagnostics\n\tctx := context.TODO()\n\tlog := c.logger(operationClientDelete)\n\n\tctx, baselog := baselogging.NewHcLogger(ctx, log)\n\tctx = baselogging.RegisterLogger(ctx, baselog)","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/hashicorp/terraform/blob/d32a084675427f5ac3f7d2868578ef8b2c1dc525/internal/backend/remote-state/s3/client.go#L222-L258","documentation":"Thrown by the S3 remote-state backend when the AWS multipart uploader fails to write the Terraform state object to the configured bucket (PutObject/Upload). The wrapped error is the raw AWS SDK v2 response, so the underlying cause (NoSuchBucket, AccessDenied, KMSNotFound, etc.) is preserved in the %w chain. This is the primary persistence path: if it fails, the new state is not saved remotely.","triggerScenarios":"The S3 manager.Uploader.Upload call at client.go:238 returns a non-nil error. Concrete triggers: bucket does not exist in the configured region, IAM principal lacks s3:PutObject on the state key, KMS key id is wrong/disabled/uses a different account, SSE-C customer key MD5 mismatches the stored object metadata, a bucket policy enforces a specific ACL/checksum the request violates, object-lock retention blocks the overwrite, or a network/transport error mid-multipart-upload.","commonSituations":"Freshly referenced bucket name typo, AWS credentials expired or scoped to the wrong account, region mismatch between the AWS client and the bucket, switched SSE configuration (KMS vs SSE-C vs AES256) without updating backend config, bucket versioning/object-lock enabled with deny-overwrite policy, running from CI with short-lived role credentials that expired mid-apply.","solutions":["Read the wrapped AWS error: run `terraform apply` again with TF_LOG=DEBUG and look for the ErrorCode/HTTP status in the chain (e.g. NoSuchBucket, AccessDenied, KMSNotFoundException) to pinpoint the cause.","Verify the bucket: `aws s3api head-bucket --bucket <bucket>` from the same environment/credentials Terraform uses; confirm it lives in the configured region.","Verify IAM permissions: ensure the effective identity has s3:PutObject on arn:aws:s3:::<bucket>/<key> (and kms:GenerateDataKey if SSE-KMS). Use the IAM policy simulator against the state key ARN.","Reconcile encryption settings: if backend config sets kms_key_id, confirm the key is enabled and the principal can use it; if using SSE-C, confirm the customer key is stable across runs.","For transient/network errors, simply retry the apply; multipart uploads are idempotent at the object key level."],"exampleFix":"# before (wrong region / missing kms)\nbackend \"s3\" {\n  bucket = \"tf-state-prod\"\n  key    = \"prod/terraform.tfstate\"\n  region = \"us-east-1\"\n}\n# after\nterraform {\n  backend \"s3\" {\n    bucket     = \"tf-state-prod\"\n    key        = \"prod/terraform.tfstate\"\n    region     = \"us-west-2\"        # match the bucket's actual region\n    kms_key_id = \"arn:aws:kms:us-west-2:111122223333:key/abcd-1234\"\n  }\n}","handlingStrategy":"retry","validationCode":"// Pre-flight: confirm the principal can write the state key before applying.\nimport (\n  \"context\"\n  \"github.com/aws/aws-sdk-go-v2/service/s3\"\n)\n\nfunc canPutState(ctx context.Context, c *s3.Client, bucket, key string) error {\n  // cheap reachability + bucket-existence check\n  if _, err := c.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: &bucket}); err != nil {\n    return fmt.Errorf(\"bucket %s not reachable: %w\", bucket, err)\n  }\n  return nil\n}\n// Call before terraform apply; surface HeadBucket failures to the operator early.","typeGuard":null,"tryCatchPattern":"// Wrap Put in a short retry for transient transport errors, surface AWS error codes.\nvar lastErr error\nfor i := 0; i < 3; i++ {\n  if _, err := uploader.Upload(ctx, input); err == nil {\n    return nil\n  } else {\n    lastErr = err\n    var apiErr smithy.APIError\n    if errors.As(err, &apiErr) {\n      switch apiErr.ErrorCode() {\n      case \"NoSuchBucket\", \"AccessDenied\", \"KMSNotFoundException\":\n        return fmt.Errorf(\"failed to upload state: %w\", err) // non-retryable\n      }\n    }\n    time.Sleep(backoff(i))\n  }\n}\nreturn fmt.Errorf(\"failed to upload state: %w\", lastErr)","preventionTips":["Run `terraform init -backend-config=...` with the same credentials Terraform will apply under, so config errors surface at init.","Keep backend encryption settings (kms_key_id / SSE-C) versioned with the config; rotate deliberately, never mid-run.","Grant the apply role a dedicated IAM policy with s3:PutObject + kms:GenerateDataKey scoped to the state key and key ARN.","Use TF_LOG=DEBUG on first apply against a new bucket to capture the exact AWS error code."],"tags":["s3","remote-state","aws","iam","upload","persistence","network","encryption"],"backgroundTag":null,"analyzedSha":"d32a084675427f5ac3f7d2868578ef8b2c1dc525","analyzedAt":"2026-08-11T18:43:52.779Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}