apache/beam · error

error parsing S3 uri

Error message

error parsing S3 uri %s: %w

What it means

Size parses the filename as an S3 URI before issuing a HeadObject; this error wraps a parseURI failure (note it uses %w so errors.Is/As unwrapping works). It indicates the filename is not a valid s3://bucket/key URI. Size returns -1 along with the error.

Solutions

1. Pass a valid s3://bucket/key URI to Size.
2. Validate/normalize URIs before calling; parseURI failures are reported via the wrapped error.
3. Use the filesystem provider matching the URI scheme.
4. Since %w is used, use errors.As to inspect the underlying parse error.

Example fix

// before
n, err := fs.Size(ctx, "my-bucket/data/file")
// after
n, err := fs.Size(ctx, "s3://my-bucket/data/file")

func validS3URI(name string) bool {
  u, err := url.Parse(name)
  return err == nil && u.Scheme == "s3" && u.Host != "" && strings.TrimPrefix(u.Path, "/") != ""
}

n, err := fs.Size(ctx, filename)
if err != nil {
  if strings.Contains(err.Error(), "error parsing S3 uri") {
    return 0, fmt.Errorf("not an s3 uri: %s", filename)
  }
  return 0, err // note %w allows errors.As unwrapping
}

When it happens

Trigger: Calling Size with a malformed URI: missing s3:// scheme, empty bucket or key, e.g. 's3://bucket' with no key or a bare local path.

Common situations: Computing input sizes for pipeline planning with paths that lost their scheme during string manipulation; local files accidentally passed to the S3 filesystem.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• error parsing S3 destination uri
• error parsing S3 source uri
• error parsing S3 uri
• Expected the endpoint to be of the form
• invalid key pattern

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ebd38574ccf76558. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:152

	return output.Body, nil
}

// OpenWrite returns a new io.WriteCloser to write contents to the file. The caller must call Close
// on the returned io.WriteCloser when done writing.
func (f *fs) OpenWrite(ctx context.Context, filename string) (io.WriteCloser, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	return newWriter(ctx, f.client, bucket, key), nil
}

// Size returns the size of the file.
func (f *fs) Size(ctx context.Context, filename string) (int64, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return -1, fmt.Errorf("error parsing S3 uri %s: %w", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.HeadObject(ctx, params)
	if err != nil {
		return -1, fmt.Errorf("error getting metadata for object %s: %w", filename, err)
	}

	if output.ContentLength != nil {
		return *output.ContentLength, nil
	}

	return -1, fmt.Errorf("content length for object %s was nil", filename)
}

View on GitHub (pinned to 12126d8942)