kubernetes/kops · error

error listing %s: %v

Error message

error listing %s: %v

What it means

S3Path.ReadDir lists objects under a key prefix using the S3 ListObjectsV2 paginator. This error is returned when any page of the paginated listing fails, wrapping the underlying AWS SDK error (permissions, missing bucket, networking). The listing is aborted and no partial result is returned.

Source

Thrown at util/pkg/vfs/s3fs.go:453

		return nil, err
	}

	prefix := p.key
	if prefix != "" && !strings.HasSuffix(prefix, "/") {
		prefix += "/"
	}
	request := &s3.ListObjectsV2Input{}
	request.Bucket = aws.String(p.bucket)
	request.Prefix = aws.String(prefix)
	request.Delimiter = aws.String("/")

	klog.V(4).Infof("Listing objects in S3 bucket %q with prefix %q", p.bucket, prefix)
	var paths []Path
	paginator := s3.NewListObjectsV2Paginator(client, request)
	for paginator.HasMorePages() {
		page, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("error listing %s: %v", p, err)
		}
		for _, o := range page.Contents {
			key := aws.ToString(o.Key)
			if key == prefix {
				// We have reports (#548 and #520) of the directory being returned as a file
				// And this will indeed happen if the directory has been created as a file,
				// which seems to happen if you use some external tools to manipulate the S3 bucket.
				// We need to tolerate that, so skip the parent directory.
				klog.V(4).Infof("Skipping read of directory: %q", key)
				continue
			}
			child := &S3Path{
				s3Context: p.s3Context,
				bucket:    p.bucket,
				key:       key,
				etag:      o.ETag,
				scheme:    p.scheme,
				sse:       p.sse,

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check the wrapped AWS error code in the message; for AccessDenied grant s3:ListBucket (and s3:ListBucket/GetBucketLocation) on the bucket to the calling principal.
2. Verify the bucket name and region in the state store configuration (kops --state s3://...) — NoSuchBucket means the bucket no longer exists or is misspelled.
3. For SlowDown/throttling errors, retry with exponential backoff or reduce listing concurrency.
4. Confirm AWS credentials and configured region match the bucket's actual region.

Example fix

// before (IAM policy)
{"Effect":"Deny","Action":"s3:ListBucket","Resource":"arn:aws:s3:::my-state-bucket"}
// after
{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":"arn:aws:s3:::my-state-bucket"}

// preflight permissions before listing
_, err := client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket)})
if err != nil { // bucket missing or no permission — fix IAM/config first }
iamSimulate := policySimulatorAllows("s3:ListBucket", bucketARN) // e.g. IAM Access Analyzer / SimulatePrincipalPolicy

func isS3ListDenied(err error) bool {
    return err != nil && (vfs.AWSErrorCode(err) == "AccessDenied" || strings.Contains(err.Error(), "AccessDenied"))
}

paths, err := s3Path.ReadDir()
if err != nil {
    switch vfs.AWSErrorCode(err) {
    case "AccessDenied":
        return fmt.Errorf("grant s3:ListBucket on %s: %w", bucket, err)
    case "NoSuchBucket":
        return fmt.Errorf("state store bucket %s does not exist", bucket)
    default:
        return err // transient — safe to retry
    }
}

When it happens

Trigger: Calling ReadDir on an S3Path when ListObjectsV2 NextPage returns an error: AccessDenied on the bucket/prefix, NoSuchBucket, throttling (SlowDown), or a network failure while iterating pages.

Common situations: IAM policy missing s3:ListBucket on the state-store bucket; bucket deleted or renamed while the kOps cluster spec still points at it; S3 request-rate throttling on very large buckets; region mismatch causing endpoint errors.

Related errors

• checking if bucket was public: %w
• failed to get bucket details for %q: %w
• failed to generate AWS IAM S3 access statements: %v
• unknown writeable path, can't apply IAM policy: %q
• getting location for bucket %q: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/147813bcb0ba42b2. Report an issue: GitHub.