kubernetes/kops · error

error writing %s: %v

Error message

error writing %s: %v

What it means

WriteFile's fallback wrap: when no ACL was attached to the PutObject request and the upload fails, the error is "error writing <path>: <underlying AWS error>". All S3-side upload failures (auth, encryption, KMS, size, network) surface here.

Source

Thrown at util/pkg/vfs/s3fs.go:354

	acl, err := p.getRequestACL(aclObj)
	if err != nil {
		return err
	}
	if acl != nil {
		request.ACL = *acl
	}

	// We don't need Content-MD5: https://github.com/aws/aws-sdk-go/issues/208

	klog.V(8).Infof("Calling S3 PutObject Bucket=%q Key=%q SSE=%q ACL=%q", p.bucket, p.key, sseLog, request.ACL)

	_, err = client.PutObject(ctx, request)
	if err != nil {
		if len(request.ACL) > 0 {
			return fmt.Errorf("error writing %s (with ACL=%q): %v", p, request.ACL, err)
		}
		return fmt.Errorf("error writing %s: %v", p, err)
	}

	return nil
}

// To prevent concurrent creates on the same file while maintaining atomicity of writes,
// we take a process-wide lock during the operation.
// Not a great approach, but fine for a single process (with low concurrency)
// TODO: should we enable versioning?
var createFileLockS3 sync.Mutex

func (p *S3Path) CreateFile(ctx context.Context, data io.ReadSeeker, acl ACL) error {
	createFileLockS3.Lock()
	defer createFileLockS3.Unlock()

	// Check if exists
	_, err := p.ReadFile(ctx)
	if err == nil {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped AWS error code; AccessDenied → grant s3:PutObject on bucket/prefix to the principal
2. If the bucket enforces SSE-KMS, ensure the caller has kms:GenerateDataKey/Decrypt on the CMK
3. If the bucket policy mandates an ACL header, set KOPS_STATE_S3_ACL (moves to the ACL branch)
4. Retry on transient errors/throttling; check region and endpoint configuration

Example fix

// before (denied by KMS)
_, err = client.PutObject(ctx, request)
// after: attach the CMK the key policy allows
request.ServerSideEncryption = types.ServerSideEncryptionAwsKms
request.SSEKMSKeyId = aws.String(allowedKeyArn)
_, err = client.PutObject(ctx, request)

_, err := s3Client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: bucket})
// 403/404 before writes signals IAM or bucket problems early

if err := p.WriteFile(ctx, data, meta, nil); err != nil {
	switch AWSErrorCode(err) {
	case "AccessDenied":
		return errStateStoreReadOnly
	case "KMS.AccessDeniedException", "InvalidKmsKeyId":
		return errKmsKeyUnusable
	default:
		return err
	}
}

When it happens

Trigger: Calling WriteFile/CreateFile on an S3Path without an ACL when PutObject fails: missing s3:PutObject permission; SSE-KMS keys the caller can't use; bucket policies requiring encryption/ACL headers not present; request timeouts on large payloads.

Common situations: Read-only state-store roles attempting writes; KMS CMK key policies denying the caller; cross-account buckets requiring bucket-owner-full-control but no ACL configured (then 3907 fires instead); dns/network issues in private clusters.

Related errors

• error writing %s (with ACL=%q): %v
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w
• error removing %d files: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/50483c939759afde. Report an issue: GitHub.