kubernetes/kops · error

error removing %d files: %w

Error message

error removing %d files: %w

What it means

RemoveAll deletes tree objects in batches of up to 1000 via the DeleteObjects (multi-delete) API. If the whole batched DeleteObjects call fails at the transport/API level, the error is wrapped as "error removing N files" where N is the batch size.

Source

Thrown at util/pkg/vfs/s3fs.go:172

	request := &s3.DeleteObjectsInput{
		Bucket: aws.String(p.bucket),
		Delete: &types.Delete{},
	}

	for len(objects) > 0 {
		// DeleteObjects can only process 1000 objects per call
		if len(objects) > 1000 {
			request.Delete.Objects = objects[:1000]
			objects = objects[1000:]
		} else {
			request.Delete.Objects = objects
			objects = nil
		}

		_, err = client.DeleteObjects(ctx, request)
		if err != nil {
			return fmt.Errorf("error removing %d files: %w", len(request.Delete.Objects), err)
		}
	}

	return nil
}

func (p *S3Path) RemoveAllVersions(ctx context.Context) error {
	client, err := p.client(ctx)
	if err != nil {
		return err
	}

	klog.V(8).Infof("removing all versions of file %s", p)

	request := &s3.ListObjectVersionsInput{
		Bucket: aws.String(p.bucket),
		Prefix: aws.String(p.key),
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check the wrapped AWS error: AccessDenied → add s3:DeleteObjects to the state-store IAM policy
2. Retry the operation on transient errors (SlowDown/RequestTimeout); batches already nil out after send
3. Verify VPC endpoint / SCP policies allow s3:DeleteObjects
4. If S3 versioning is enabled, consider RemoveAllVersions instead to purge all versions and markers

Example fix

// before
_, err = client.DeleteObjects(ctx, request)
if err != nil { return fmt.Errorf("error removing %d files: %w", len(request.Delete.Objects), err) }
// after (caller): retry transient batch failures
err = wait.ExponentialBackoff(backoff, func() (bool, error) {
	_, err := retryRemoveAll(ctx)
	return err == nil, nil
})

// Verify multi-delete permission with a dry-run style call on a scratch key
_, err := s3Client.DeleteObjects(ctx, &s3.DeleteObjectsInput{
	Bucket: bucket,
	Delete: &types.Delete{Objects: []types.ObjectIdentifier{{Key: aws.String("__perm_test__")}}},
})

err := wait.ExponentialBackoff(retryBackoff, func() (bool, error) {
	err := root.RemoveAll(ctx)
	if err != nil && (AWSErrorCode(err) == "SlowDown" || AWSErrorCode(err) == "InternalError") {
		return false, nil
	}
	return err == nil, err
})

When it happens

Trigger: Calling RemoveAll on an S3Path when DeleteObjects fails: IAM policy grants DeleteObject but not DeleteObjects on the bucket; more than 1000 keys per request (impossible — batches are chunked, so this is API-level failure); throttling or network failure during the batch call.

Common situations: IAM policies missing s3:DeleteObjects (multi-object delete) while allowing single-object deletes; large cluster deletions triggering S3 throttling (SlowDown); VPC endpoint policy blocking the DeleteObjects action.

Related errors

• error removing %d file/marker versions: %v
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w
• error listing all versions of file %s: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/cd288fb3dfa70947. Report an issue: GitHub.