apache/beam · error

error getting metadata for object

Error message

error getting metadata for object %s: %v

What it means

This error is returned by the S3 filesystem's LastModified method when the AWS SDK HeadObject call fails to fetch object metadata. It wraps the underlying AWS error with the full filename for context. It means the object could not be found, is inaccessible, or the S3 request itself failed.

Solutions

1. Verify the object exists (correct bucket and key, no typos) via the AWS console or `aws s3 ls`.
2. Check IAM permissions: the credentials must allow s3:HeadObject on the bucket/key.
3. Confirm AWS credentials and region configuration (env vars, ~/.aws, or SDK config).
4. Inspect the wrapped cause (%v) for specific codes like NotFound, AccessDenied, or NoCredentialProviders and fix accordingly.

Example fix

// before: assuming object exists
lastModified, err := fsys.LastModified(ctx, "s3://my-bucket/data/2026/output.txt")

// after: parse/verify URI and handle missing object
if !strings.HasPrefix(filename, "s3://") { return fmt.Errorf("not an s3 path: %s", filename) }
lastModified, err := fsys.LastModified(ctx, filename)
if err != nil {
    var nfe *types.NotFound
    if errors.As(err, &nfe) { return nil } // treat as absent
    return err
}

if !strings.HasPrefix(filename, "s3://") { return fmt.Errorf("not an s3 URI: %s", filename) }
bucket, key, err := parseS3URI(filename)
if err != nil { return err }
_, err = s3Client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
if err != nil { /* object likely absent or inaccessible */ }

func isS3Path(p string) bool { return strings.HasPrefix(p, "s3://") && len(strings.TrimPrefix(p, "s3://")) > 0 }

t, err := fsys.LastModified(ctx, filename)
if err != nil {
    var nfe *types.NotFound
    if errors.As(err, &nfe) { return nil, ErrObjectAbsent }
    if strings.Contains(err.Error(), "AccessDenied") { return nil, ErrPermission }
    return fmt.Errorf("LastModified(%s): %w", filename, err)
}

When it happens

Trigger: Calling fs.LastModified(ctx, filename) when the bucket/key does not exist, the caller lacks s3:HeadObject permission, the region is wrong, or the AWS credentials are missing/expired.

Common situations: Checking a file before reading it when the object was deleted by another process; typo'd bucket or key; IAM policy missing s3:HeadObject; misconfigured AWS_REGION; expired session credentials in CI.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• error copying object
• error deleting object
• error getting metadata for object
• error getting object
• error listing object keys

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ab25f3b8fa23f2ae. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:184

	}

	return -1, fmt.Errorf("content length for object %s was nil", filename)
}

// LastModified returns the time at which the file was last modified.
func (f *fs) LastModified(ctx context.Context, filename string) (time.Time, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return time.Time{}, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.HeadObject(ctx, params)
	if err != nil {
		return time.Time{}, fmt.Errorf("error getting metadata for object %s: %v", filename, err)
	}

	return aws.ToTime(output.LastModified), err
}

// Remove removes the file from the filesystem.
func (f *fs) Remove(ctx context.Context, filename string) error {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.DeleteObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	if _, err = f.client.DeleteObject(ctx, params); err != nil {
		return fmt.Errorf("error deleting object %s: %v", filename, err)

View on GitHub (pinned to 12126d8942)