kubernetes/kops · error

error listing all versions of file %s: %v

Error message

error listing all versions of file %s: %v

What it means

RemoveAllVersions lists every version and delete marker of an object using the ListObjectVersions paginator so it can delete them all. If any page of the pagination fails, the error is wrapped as "error listing all versions of file <path>".

Source

Thrown at util/pkg/vfs/s3fs.go:198

	client, err := p.client(ctx)
	if err != nil {
		return err
	}

	klog.V(8).Infof("removing all versions of file %s", p)

	request := &s3.ListObjectVersionsInput{
		Bucket: aws.String(p.bucket),
		Prefix: aws.String(p.key),
	}

	var versions []types.ObjectVersion
	var deleteMarkers []types.DeleteMarkerEntry
	paginator := s3.NewListObjectVersionsPaginator(client, request)
	for paginator.HasMorePages() {
		page, err := paginator.NextPage(ctx)
		if err != nil {
			return fmt.Errorf("error listing all versions of file %s: %v", p, err)
		}
		for _, version := range page.Versions {
			if aws.ToString(version.Key) == p.key {
				versions = append(versions, version)
			}
		}
		for _, marker := range page.DeleteMarkers {
			if aws.ToString(marker.Key) == p.key {
				deleteMarkers = append(deleteMarkers, marker)
			}
		}
	}

	if len(versions) == 0 && len(deleteMarkers) == 0 {
		return os.ErrNotExist
	}

	var objects []types.ObjectIdentifier

View on GitHub (pinned to 4c8573c808)

Solutions

1. Grant s3:ListBucketVersions (and s3:ListBucket) on the bucket to the operating principal
2. Retry on transient errors; pagination resumes by rerunning the operation
3. Reduce version sprawl with lifecycle rules so listings stay small
4. Check bucket region/client config if the error indicates wrong endpoint

Example fix

// before
if err != nil { return fmt.Errorf("error listing all versions of file %s: %v", p, err) }
// after (caller): pre-check permission/availability
_, err := s3Client.ListObjectVersions(ctx, &s3.ListObjectVersionsInput{Bucket: bucket, MaxKeys: aws.Int32(1)})
if err != nil { return fmt.Errorf("state store not listable: %w", err) }

_, err := s3Client.ListObjectVersions(ctx, &s3.ListObjectVersionsInput{Bucket: bucket, MaxKeys: aws.Int32(1)})
if err != nil { return fmt.Errorf("state store not listable (need s3:ListBucketVersions): %w", err) }

err := wait.ExponentialBackoff(listBackoff, func() (bool, error) {
	return p.RemoveAllVersions(ctx) == nil, nil
})

When it happens

Trigger: Calling RemoveAllVersions on a versioning-enabled S3 bucket when ListObjectVersions fails: IAM policy missing s3:ListBucketVersions; throttling on buckets with very many versions; network failure mid-pagination.

Common situations: State-store buckets with versioning enabled (recommended by kOps) and restrictive IAM roles; buckets with huge version history causing slow/paged listing that hits limits.

Related errors

• error removing %d file/marker versions: %v
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• checking if bucket was public: %w
• error listing IAM instance profiles: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/95300eb9bc76f029. Report an issue: GitHub.