apache/beam · error

error getting metadata for object %s: %w

Error message

error getting metadata for object %s: %w

What it means

Size issues a HeadObject request to fetch ContentLength and wraps any HeadObject failure with this message (using %w for unwrapping). The URI parsed fine but the metadata request failed — typically object not found, access denied, wrong region, or network error. Size returns -1 with this error.

Source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:161

	}

	return newWriter(ctx, f.client, bucket, key), nil
}

// Size returns the size of the file.
func (f *fs) Size(ctx context.Context, filename string) (int64, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return -1, fmt.Errorf("error parsing S3 uri %s: %w", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.HeadObject(ctx, params)
	if err != nil {
		return -1, fmt.Errorf("error getting metadata for object %s: %w", filename, err)
	}

	if output.ContentLength != nil {
		return *output.ContentLength, nil
	}

	return -1, fmt.Errorf("content length for object %s was nil", filename)
}

// LastModified returns the time at which the file was last modified.
func (f *fs) LastModified(ctx context.Context, filename string) (time.Time, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return time.Time{}, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the object exists and the key is exact (case-sensitive).
2. Check IAM allows s3:GetObject/HeadObject on the key.
3. Align the client region with the bucket region.
4. Retry transient network failures with backoff; inspect the wrapped AWS error code.

Example fix

// before
n, err := fs.Size(ctx, uri) // assume success
// after
n, err := fs.Size(ctx, uri)
if err != nil {
  var ae smithy.APIError
  if errors.As(err, &ae) && ae.ErrorCode() == "NotFound" {
    // handle missing object
  }
}

// verify readability before requesting size
_, err := f.client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(b), Key: aws.String(k)})
if err != nil { return fmt.Errorf("cannot stat s3://%s/%s: %w", b, k, err) }

n, err := fs.Size(ctx, uri)
if err != nil {
  var ae smithy.APIError
  if errors.As(err, &ae) {
    switch ae.ErrorCode() {
    case "NotFound", "NoSuchKey": return 0, fmt.Errorf("object missing: %s", uri)
    default:
      if strings.Contains(ae.Error(), "throttl") { return retryWithBackoff(ctx) }
    }
  }
  return 0, err
}

When it happens

Trigger: Calling Size on s3://bucket/key where HeadObject returns NoSuchKey/NoSuchBucket/403/404, credentials lack s3:GetObject (HeadObject requires it), or a network failure occurs.

Common situations: Sizing inputs that were later deleted or renamed; region mismatch between client and bucket; IAM policies granting ListBucket but not GetObject; intermittent network faults in long-running workers.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• str(e) (wrapped S3 client error message)
• str(e)
• error listing object keys: %v
• error retrieving page: %v
• error getting object %s: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ec25f87d48f1f3bc. Report an issue: GitHub.