kubernetes/kops · error

error removing %d file/marker versions: %v

Error message

error removing %d file/marker versions: %v

What it means

After collecting all versions and delete markers, RemoveAllVersions removes them in batched DeleteObjects calls. If the call itself fails, this error reports how many file/marker versions were in the failed request batch.

Source

Thrown at util/pkg/vfs/s3fs.go:253

		request := &s3.DeleteObjectsInput{
			Bucket: aws.String(p.bucket),
			Delete: &types.Delete{},
		}

		// DeleteObjects can only process 1000 objects per call
		if len(objects) > 1000 {
			request.Delete.Objects = objects[:1000]
			objects = objects[1000:]
		} else {
			request.Delete.Objects = objects
			objects = nil
		}

		klog.V(8).Infof("removing %d file/marker versions\n", len(request.Delete.Objects))

		deleteResult, err := client.DeleteObjects(ctx, request)
		if err != nil {
			return fmt.Errorf("error removing %d file/marker versions: %v", len(request.Delete.Objects), err)
		}
		if len(deleteResult.Errors) > 0 {
			return fmt.Errorf("error removing file/marker versions: %v", deleteResult.Errors)
		}
	}

	return nil
}

func (p *S3Path) Join(relativePath ...string) Path {
	args := []string{p.key}
	args = append(args, relativePath...)
	joined := path.Join(args...)
	return &S3Path{
		s3Context: p.s3Context,
		bucket:    p.bucket,
		key:       joined,
		scheme:    p.scheme,

View on GitHub (pinned to 4c8573c808)

Solutions

1. Add s3:DeleteObjectVersion to the IAM policy for the state-store bucket
2. Disable MFA-delete or use MFA-authenticated deletion if MFA-delete is on
3. Retry the operation on throttling/transient errors
4. Verify no bucket policy/SCP denies DeleteObjectVersion for the principal

Example fix

// before
_, err := client.DeleteObjects(ctx, request)
if err != nil { return fmt.Errorf("error removing %d file/marker versions: %v", len(request.Delete.Objects), err) }
// after (caller): retry with backoff
backoff := wait.Backoff{Steps: 5, Duration: time.Second, Factor: 2}
err = wait.ExponentialBackoff(backoff, func() (bool, error) { return retryRemoveAllVersions(ctx) == nil, nil })

// Ensure version deletion is permitted before bulk cleanup
_, err := s3Client.DeleteObjectVersion(ctx, &s3.DeleteObjectVersionInput{Bucket: bucket, Key: k, VersionId: v})
// AccessDenied ⇒ s3:DeleteObjectVersion missing

if err := p.RemoveAllVersions(ctx); err != nil {
	if AWSErrorCode(err) == "AccessDenied" { return errMissingDeleteObjectVersion }
	// else retry transient codes
}

When it happens

Trigger: Calling RemoveAllVersions on a versioned bucket when DeleteObjects (with VersionId set per object) fails: IAM missing s3:DeleteObjectVersion; throttling; >1000 versions chunked correctly but a chunk fails on network/API error.

Common situations: Roles allowed to delete current objects but not specific old versions (s3:DeleteObjectVersion absent); MFA-delete enabled buckets; transient SlowDown during large multi-version cleanups.

Related errors

• error removing %d files: %w
• error listing all versions of file %s: %v
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/f5df4cfc13787463. Report an issue: GitHub.