kubernetes/kops · error

error writing %s (with ACL=%q): %v

Error message

error writing %s (with ACL=%q): %v

What it means

WriteFile uploads the object with PutObject; when an ACL was attached to the request and the upload fails, the error is wrapped as "error writing <path> (with ACL=<acl>)" so the failing ACL is visible. The %v carries the underlying AWS error (most commonly AccessDenied).

Source

Thrown at util/pkg/vfs/s3fs.go:352

	var sseLog string
	request.ServerSideEncryption, sseLog, _ = p.getServerSideEncryption(ctx)

	acl, err := p.getRequestACL(aclObj)
	if err != nil {
		return err
	}
	if acl != nil {
		request.ACL = *acl
	}

	// We don't need Content-MD5: https://github.com/aws/aws-sdk-go/issues/208

	klog.V(8).Infof("Calling S3 PutObject Bucket=%q Key=%q SSE=%q ACL=%q", p.bucket, p.key, sseLog, request.ACL)

	_, err = client.PutObject(ctx, request)
	if err != nil {
		if len(request.ACL) > 0 {
			return fmt.Errorf("error writing %s (with ACL=%q): %v", p, request.ACL, err)
		}
		return fmt.Errorf("error writing %s: %v", p, err)
	}

	return nil
}

// To prevent concurrent creates on the same file while maintaining atomicity of writes,
// we take a process-wide lock during the operation.
// Not a great approach, but fine for a single process (with low concurrency)
// TODO: should we enable versioning?
var createFileLockS3 sync.Mutex

func (p *S3Path) CreateFile(ctx context.Context, data io.ReadSeeker, acl ACL) error {
	createFileLockS3.Lock()
	defer createFileLockS3.Unlock()

	// Check if exists

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped AWS error: AccessDenied with ACL usually means the bucket policy's s3:x-amz-acl condition rejected the request
2. Set KOPS_STATE_S3_ACL to a valid canned ACL the policy allows (e.g. bucket-owner-full-control for cross-account state stores)
3. Verify the canned ACL spelling (private, public-read, bucket-owner-full-control, ...)
4. For cross-account buckets, ensure the bucket policy grants the kOps principal PutObject with the required ACL condition

Example fix

// before
export KOPS_STATE_S3_ACL=bucket-owner-read
// after (allowed by the bucket policy)
export KOPS_STATE_S3_ACL=bucket-owner-full-control

// Confirm the canned ACL will be accepted: check bucket policy condition
pol, _ := s3Client.GetBucketPolicy(ctx, &s3.GetBucketPolicyInput{Bucket: bucket})
allowed := strings.Contains(aws.ToString(pol.Policy), "s3:x-amz-acl") // inspect conditions

func validCannedACL(s string) bool {
	switch types.ObjectCannedACL(s) {
	case types.ObjectCannedACLPrivate, types.ObjectCannedACLPublicRead,
		types.ObjectCannedACLBucketOwnerFullControl, types.ObjectCannedACLBucketOwnerRead:
		return true
	}
	return false
}

if err := p.WriteFile(ctx, data, meta, acl); err != nil {
	if AWSErrorCode(err) == "AccessDenied" {
		return fmt.Errorf("PutObject rejected with ACL %q — align KOPS_STATE_S3_ACL with bucket policy: %w", acl, err)
	}
	return err
}

When it happens

Trigger: Calling WriteFile/CreateFile on an S3Path with an ACL set when PutObject fails: the ACL string is invalid/unrecognized, or the principal is not allowed to PutObject with that canned ACL (s3:PutObject with s3:x-amz-acl condition).

Common situations: KOPS_STATE_S3_ACL set to a value the bucket policy forbids (e.g. bucket-owner-full-control without cross-account permission); typo'd canned ACL names; cross-account state stores missing bucket-owner ACL grants.

Related errors

• error writing %s: %v
• failed to get grant for key %q in bucket %q: %w
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/1d71d1a148d33f82. Report an issue: GitHub.