{"record":{"id":"50483c939759afde","repo":"kubernetes/kops","slug":"error-writing-s-v-50483c","errorCode":null,"errorMessage":"error writing %s: %v","messagePattern":"error writing (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/s3fs.go","lineNumber":354,"sourceCode":"\n\tacl, err := p.getRequestACL(aclObj)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif acl != nil {\n\t\trequest.ACL = *acl\n\t}\n\n\t// We don't need Content-MD5: https://github.com/aws/aws-sdk-go/issues/208\n\n\tklog.V(8).Infof(\"Calling S3 PutObject Bucket=%q Key=%q SSE=%q ACL=%q\", p.bucket, p.key, sseLog, request.ACL)\n\n\t_, err = client.PutObject(ctx, request)\n\tif err != nil {\n\t\tif len(request.ACL) > 0 {\n\t\t\treturn fmt.Errorf(\"error writing %s (with ACL=%q): %v\", p, request.ACL, err)\n\t\t}\n\t\treturn fmt.Errorf(\"error writing %s: %v\", p, err)\n\t}\n\n\treturn nil\n}\n\n// To prevent concurrent creates on the same file while maintaining atomicity of writes,\n// we take a process-wide lock during the operation.\n// Not a great approach, but fine for a single process (with low concurrency)\n// TODO: should we enable versioning?\nvar createFileLockS3 sync.Mutex\n\nfunc (p *S3Path) CreateFile(ctx context.Context, data io.ReadSeeker, acl ACL) error {\n\tcreateFileLockS3.Lock()\n\tdefer createFileLockS3.Unlock()\n\n\t// Check if exists\n\t_, err := p.ReadFile(ctx)\n\tif err == nil {","sourceCodeStart":336,"sourceCodeEnd":372,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/s3fs.go#L336-L372","documentation":"WriteFile's fallback wrap: when no ACL was attached to the PutObject request and the upload fails, the error is \"error writing <path>: <underlying AWS error>\". All S3-side upload failures (auth, encryption, KMS, size, network) surface here.","triggerScenarios":"Calling WriteFile/CreateFile on an S3Path without an ACL when PutObject fails: missing s3:PutObject permission; SSE-KMS keys the caller can't use; bucket policies requiring encryption/ACL headers not present; request timeouts on large payloads.","commonSituations":"Read-only state-store roles attempting writes; KMS CMK key policies denying the caller; cross-account buckets requiring bucket-owner-full-control but no ACL configured (then 3907 fires instead); dns/network issues in private clusters.","solutions":["Read the wrapped AWS error code; AccessDenied → grant s3:PutObject on bucket/prefix to the principal","If the bucket enforces SSE-KMS, ensure the caller has kms:GenerateDataKey/Decrypt on the CMK","If the bucket policy mandates an ACL header, set KOPS_STATE_S3_ACL (moves to the ACL branch)","Retry on transient errors/throttling; check region and endpoint configuration"],"exampleFix":"// before (denied by KMS)\n_, err = client.PutObject(ctx, request)\n// after: attach the CMK the key policy allows\nrequest.ServerSideEncryption = types.ServerSideEncryptionAwsKms\nrequest.SSEKMSKeyId = aws.String(allowedKeyArn)\n_, err = client.PutObject(ctx, request)","handlingStrategy":"try-catch","validationCode":"_, err := s3Client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: bucket})\n// 403/404 before writes signals IAM or bucket problems early","typeGuard":null,"tryCatchPattern":"if err := p.WriteFile(ctx, data, meta, nil); err != nil {\n\tswitch AWSErrorCode(err) {\n\tcase \"AccessDenied\":\n\t\treturn errStateStoreReadOnly\n\tcase \"KMS.AccessDeniedException\", \"InvalidKmsKeyId\":\n\t\treturn errKmsKeyUnusable\n\tdefault:\n\t\treturn err\n\t}\n}","preventionTips":["Verify write access with a canary object before real state writes","Ensure KMS key policies allow the writer principal when SSE-KMS is enforced","Confirm the bucket region matches the SDK region configuration","Enable SDK retryer defaults for transient PutObject failures"],"tags":["aws","s3","write","kms","iam"],"backgroundTag":"s3-putobject-access-denied","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}