apache/beam · error

error parsing S3 uri

Error message

error parsing S3 uri %s: %v

What it means

OpenRead parses the filename as an S3 URI before downloading; this error wraps a parseURI failure. It means the filename string is not a valid s3://bucket/key URI. The offending filename and underlying parse error are both included in the message.

Solutions

1. Pass a well-formed URI of the form s3://bucket/key.
2. Check which filesystem provider the pipeline resolved for this path — use the matching one.
3. Normalize/validate the filename before calling OpenRead (see validation code).
4. Inspect the wrapped parse error to see which URI component failed.

Example fix

// before
r, err := fs.OpenRead(ctx, "/data/file.txt")
// after
r, err := fs.OpenRead(ctx, "s3://my-bucket/data/file.txt")

func validS3URI(name string) bool {
  u, err := url.Parse(name)
  return err == nil && u.Scheme == "s3" && u.Host != "" && strings.TrimPrefix(u.Path, "/") != ""
}

r, err := fs.OpenRead(ctx, filename)
if err != nil {
  if strings.Contains(err.Error(), "error parsing S3 uri") {
    return fmt.Errorf("bad S3 path %q, expected s3://bucket/key", filename)
  }
  return fmt.Errorf("open failed: %w", err)
}

When it happens

Trigger: Calling OpenRead with a filename lacking the s3:// scheme, with an empty bucket or key, or otherwise malformed (e.g. 's3:///no-key', 'gs://bucket/obj', or a local path).

Common situations: Mixing up filesystem providers (passing GCS or local paths to the S3 filesystem); a resource name missing its scheme after path joining; typos in 's3://' prefix.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• error parsing S3 destination uri
• error parsing S3 source uri
• error parsing S3 uri
• Expected the endpoint to be of the form
• invalid key pattern

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/01cd26d111d556e0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:122

			if err != nil {
				return nil, fmt.Errorf("invalid key pattern: %s", keyPattern)
			}

			if match {
				objects = append(objects, key)
			}
		}
	}

	return objects, nil
}

// OpenRead returns a new io.ReadCloser to read contents from the file. The caller must call Close
// on the returned io.ReadCloser when done reading.
func (f *fs) OpenRead(ctx context.Context, filename string) (io.ReadCloser, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.GetObject(ctx, params)
	if err != nil {
		return nil, fmt.Errorf("error getting object %s: %v", filename, err)
	}

	return output.Body, nil
}

// OpenWrite returns a new io.WriteCloser to write contents to the file. The caller must call Close
// on the returned io.WriteCloser when done writing.
func (f *fs) OpenWrite(ctx context.Context, filename string) (io.WriteCloser, error) {
	bucket, key, err := parseURI(filename)

View on GitHub (pinned to 12126d8942)