What causes "error parsing S3 uri: %v" in apache/beam?

Ensure the glob starts with 's3://' and includes bucket and key pattern, e.g. 's3://bucket/prefix/*.json'.

error parsing S3 uri: %v

Error message

error parsing S3 uri: %v

What it means

S3 List first parses the glob into a bucket and key pattern via parseURI; if the string is not a well-formed S3 URI (no bucket, bad scheme, unparseable form), the parse error is wrapped as 'error parsing S3 uri: %v'. The request never reaches AWS.

Source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:61

	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		panic(fmt.Sprintf("error loading AWS config: %v", err))
	}

	client := s3.NewFromConfig(cfg)
	return &fs{client: client}
}

// Close closes the filesystem.
func (f *fs) Close() error {
	return nil
}

// List returns a slice of the files in the filesystem that match the glob pattern.
func (f *fs) List(ctx context.Context, glob string) ([]string, error) {
	bucket, keyPattern, err := parseURI(glob)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri: %v", err)
	}

	keys, err := f.listObjectKeys(ctx, bucket, keyPattern)
	if err != nil {
		return nil, fmt.Errorf("error listing object keys: %v", err)
	}

	if len(keys) == 0 {
		return nil, nil
	}

	uris := make([]string, len(keys))
	for i, key := range keys {
		uris[i] = makeURI(bucket, key)
	}

	return uris, nil
}

View on GitHub (pinned to 12126d8942)

Solutions

Ensure the glob starts with 's3://' and includes bucket and key pattern, e.g. 's3://bucket/prefix/*.json'.
Trim whitespace and validate the URI shape before calling List.
Fix the config/flag source producing the malformed URI.
Check the wrapped inner error to see exactly which URI component parseURI rejected.

Example fix

// before
files, err := fs.List(ctx, "mybucket/data/*.json") // missing scheme
// after
files, err := fs.List(ctx, "s3://mybucket/data/*.json")

Defensive patterns

Strategy: validation

Validate before calling

func validS3URI(u string) bool {
	rest, ok := strings.CutPrefix(u, "s3://")
	if !ok { return false }
	bucket, key, _ := strings.Cut(rest, "/")
	return bucket != "" && key != ""
}

Try / catch

if _, err := fs.List(ctx, glob); err != nil && strings.Contains(err.Error(), "error parsing S3 uri") {
	return fmt.Errorf("expected s3://bucket/key form, got %q: %w", glob, err)
}

Prevention

Always build S3 paths as s3://bucket/pattern.
Trim whitespace from config values before use.
Validate URIs at config-load time, not deep in the pipeline.
Don't pass plain paths or https URLs into the S3 filesystem.

When it happens

Trigger: Calling fs.List(ctx, glob) with a string that parseURI rejects — e.g. missing 's3://' scheme, empty bucket, or a path with no key component where one is required.

Common situations: Passing a plain path ('/data/file') or an HTTP URL instead of an s3:// URI; concatenating bucket and key incorrectly; config values with typos or stray whitespace.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fd0db3d88c4ddc4f. Report an issue: GitHub.