What causes "error retrieving page: %v" in apache/beam?

Inspect the wrapped AWS error; handle smithy APIError codes like AccessDenied or SlowDown.

error retrieving page: %v

Error message

error retrieving page: %v

What it means

listObjectKeys fetches S3 object listings page by page with ListObjectsV2Paginator; this error wraps a failure to retrieve any single page. It indicates the S3 API rejected or failed on a pagination request, typically auth, permission, throttling, or network problems. It is then further wrapped by List's 'error listing object keys' message.

Source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:98

// listObjectKeys returns a slice of the keys in the bucket that match the key pattern.
func (f *fs) listObjectKeys(
	ctx context.Context,
	bucket string,
	keyPattern string,
) ([]string, error) {
	prefix := fsx.GetPrefix(keyPattern)
	params := &s3.ListObjectsV2Input{
		Bucket: aws.String(bucket),
		Prefix: aws.String(prefix),
	}
	paginator := s3.NewListObjectsV2Paginator(f.client, params)

	var objects []string
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("error retrieving page: %v", err)
		}

		for _, object := range output.Contents {
			key := aws.ToString(object.Key)
			match, err := filepath.Match(keyPattern, key)
			if err != nil {
				return nil, fmt.Errorf("invalid key pattern: %s", keyPattern)
			}

			if match {
				objects = append(objects, key)
			}
		}
	}

	return objects, nil
}

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the wrapped AWS error; handle smithy APIError codes like AccessDenied or SlowDown.
Retry with backoff on throttling; consider retryer options on the S3 client config.
Refresh or fix credentials if session tokens expired.
Check s3:ListBucket permission for the exact bucket.

Example fix

// before
client := s3.NewFromConfig(cfg)
// after
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
  o.Retryer = retry.AddWithMaxBackoffDelay(retry.NewStandard(), 5*time.Second)
})

Defensive patterns

Strategy: retry

Validate before calling

// check permissions/creds before paginating
creds, err := awsConfig.Credentials.Retrieve(ctx)
if err != nil { return err }
// probe with a single small listing
p := s3.NewListObjectsV2Paginator(f.client, &s3.ListObjectsV2Input{Bucket: aws.String(bucket), MaxKeys: aws.Int32(1)})
if _, err := p.NextPage(ctx); err != nil { return err }

Try / catch

_, err := paginator.NextPage(ctx)
if err != nil {
  var re *retry.MaxAttemptsError
  if errors.As(err, &re) || isThrottle(err) {
    time.Sleep(backoff); continue
  }
  return fmt.Errorf("page fetch failed: %w", err)
}

Prevention

Use a standard retryer with exponential backoff on the S3 client.
Refresh STS/temporary credentials before long iterations.
Probe with MaxKeys=1 to validate access before large scans.
Watch for SlowDown errors on very large buckets and shard prefixes.

When it happens

Trigger: paginator.NextPage(ctx) returns an error while iterating HasMorePages() — e.g. AccessDenied on ListObjectsV2, slow-down throttling on large buckets, expired session credentials mid-iteration, or network interruption.

Common situations: Large buckets requiring many pages hit S3 throttling (SlowDown); temporary credentials (STS) expire during iteration; IAM role not yet propagated in a fresh environment.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5c109c1e97992080. Report an issue: GitHub.