kubernetes/kops · error

error deleting %s: %v

Error message

error deleting %s: %v

What it means

S3Path.Remove deletes a single object from S3 via DeleteObject. When the S3 API call fails for any reason (permissions, networking, bucket state), the error is wrapped as "error deleting <path>: <underlying error>". The TODO in the source notes that a not-exists response is not yet translated to os.ErrNotExist, so even a missing object surfaces through this wrapper.

Source

Thrown at util/pkg/vfs/s3fs.go:124

}

func (p *S3Path) Remove(ctx context.Context) error {
	client, err := p.client(ctx)
	if err != nil {
		return err
	}

	klog.V(8).Infof("removing file %s", p)

	request := &s3.DeleteObjectInput{}
	request.Bucket = aws.String(p.bucket)
	request.Key = aws.String(p.key)

	_, err = client.DeleteObject(ctx, request)
	if err != nil {
		// TODO: Check for not-exists, return os.NotExist

		return fmt.Errorf("error deleting %s: %v", p, err)
	}

	return nil
}

func (p *S3Path) RemoveAll(ctx context.Context) error {
	client, err := p.client(ctx)
	if err != nil {
		return err
	}

	tree, err := p.ReadTree(ctx)
	if err != nil {
		return err
	}

	objects := make([]types.ObjectIdentifier, len(tree))
	for i := range tree {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped %v error to identify the AWS failure code (AccessDenied, NoSuchBucket, etc.)
2. Verify the credentials/role used have s3:DeleteObject permission on the state-store bucket/prefix
3. Confirm KOPS_STATE_STORE points at the correct bucket and region (AWS_REGION / bucket region match)
4. Re-run after fixing transient network/throttling issues; consider AWS SDK retries

Example fix

// before (library TODO): not-exists is not mapped
return fmt.Errorf("error deleting %s: %v", p, err)
// after (caller-side handling)
if err := ctx.Remove(p); err != nil {
	if AWSErrorCode(err) == "NoSuchKey" || strings.Contains(err.Error(), "NotFound") {
		return nil // already gone
	}
	return err
}

// Pre-check delete permission minimally
_, err := s3Client.DeleteObjectTagging(ctx, &s3.DeleteObjectTaggingInput{Bucket: bucket, Key: aws.String(key)})
// treat AccessDenied as "will fail" before attempting Remove

err := vfs.Context.Remove(p)
if err != nil {
	if AWSErrorCode(err) == "AccessDenied" {
		return fmt.Errorf("no s3:DeleteObject on %s: %w", p, err)
	}
	if errors.Is(err, context.DeadlineExceeded) { /* retry */ }
	return err
}

When it happens

Trigger: Calling vfs context Remove() on an S3Path when: the IAM credentials lack s3:DeleteObject on the key; the AWS request fails (throttling, network, expired credentials); the bucket is in a region/account the client cannot reach; the object key is invalid or the bucket does not exist.

Common situations: Running `kops delete cluster` or state-store cleanup with a read-only state store policy; corporate proxy or VPC endpoint blocking S3; expired STS session tokens during long operations; typos in KOPS_STATE_STORE bucket name.

Related errors

• checking if bucket was public: %w
• getting location for bucket %q: %w
• error listing %s: %v
• failed to get bucket details for %q: %w
• error removing cluster from state store: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/ea6eb30b28f3dfb2. Report an issue: GitHub.