What causes "error listing object keys: %v" in apache/beam?

Fix the underlying error shown in the wrapped %v text (often an AWS SDK smithy operation error).

error listing object keys: %v

Error message

error listing object keys: %v

What it means

List() wraps any failure from listObjectKeys (which enumerates S3 object keys via ListObjectsV2) with this message. It means the S3 listing operation failed, e.g. due to credentials, bucket access, or API errors. The original error is embedded via %v so check the wrapped text for the root cause.

Source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:66

	client := s3.NewFromConfig(cfg)
	return &fs{client: client}
}

// Close closes the filesystem.
func (f *fs) Close() error {
	return nil
}

// List returns a slice of the files in the filesystem that match the glob pattern.
func (f *fs) List(ctx context.Context, glob string) ([]string, error) {
	bucket, keyPattern, err := parseURI(glob)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri: %v", err)
	}

	keys, err := f.listObjectKeys(ctx, bucket, keyPattern)
	if err != nil {
		return nil, fmt.Errorf("error listing object keys: %v", err)
	}

	if len(keys) == 0 {
		return nil, nil
	}

	uris := make([]string, len(keys))
	for i, key := range keys {
		uris[i] = makeURI(bucket, key)
	}

	return uris, nil
}

// listObjectKeys returns a slice of the keys in the bucket that match the key pattern.
func (f *fs) listObjectKeys(
	ctx context.Context,
	bucket string,

View on GitHub (pinned to 12126d8942)

Solutions

Fix the underlying error shown in the wrapped %v text (often an AWS SDK smithy operation error).
Verify AWS credentials are available (env vars, ~/.aws/credentials, or instance role).
Confirm the bucket exists and the client region matches the bucket region.
Grant the caller IAM permission s3:ListBucket on the bucket.

Example fix

// before
client := s3.NewFromConfig(cfg) // cfg.Region may be empty
// after
awsCfg, _ := config.LoadDefaultConfig(ctx, config.WithRegion("us-east-1"))
client := s3.NewFromConfig(awsCfg)

Defensive patterns

Strategy: retry

Validate before calling

// ensure credentials & bucket reachable before List
sess := credentials.NewEnvCredentials()
c, err := sess.Retrieve(ctx)
if err != nil { return fmt.Errorf("no AWS credentials: %w", err) }
if _, err := f.client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket)}); err != nil {
  return fmt.Errorf("bucket %s unreachable: %w", bucket, err)
}

Try / catch

keys, err := fs.List(ctx, glob)
if err != nil {
  if isRetryable(err) { // e.g. throttling / network
     return retryWithBackoff(ctx, func() error { _, err = fs.List(ctx, glob); return err })
  }
  return fmt.Errorf("s3 list failed for %s: %w", glob, err)
}

Prevention

Configure AWS credentials via a standard chain before running pipelines.
Verify s3:ListBucket IAM permission for the target bucket.
Match client region to bucket region.
Enable SDK retryer with backoff for throttling.

When it happens

Trigger: Calling List(ctx, glob) on the S3 filesystem when the ListObjectsV2 call fails: missing/invalid AWS credentials, nonexistent bucket, insufficient s3:ListBucket permission, wrong region, or network failure.

Common situations: Default AWS credential chain not configured in the environment; IAM policy lacking ListBucket; bucket in a different region than the client config; transient network issues during long Beam pipelines.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d4af03b9a29d0d4e. Report an issue: GitHub.