What causes "scheme must be 's3'" in apache/beam?

Use proper s3 URIs: s3:// / (e.g. s3://my-bucket/data/file.json).

scheme must be 's3'

Error message

scheme must be 's3'

What it means

The s3 filesystem package parses URIs before any S3 operation (List, OpenRead, OpenWrite, Size, LastModified, Remove) and requires the URI scheme to be exactly 's3'. Any other scheme (http, file, gs, plain paths) is rejected because this driver only handles s3:// URLs.

Source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/util.go:32

// limitations under the License.

package s3

import (
	"errors"
	"fmt"
	"net/url"
)

// parseURI deconstructs the S3 uri in the format 's3://bucket/key' to (bucket, key)
func parseURI(uri string) (string, string, error) {
	parsed, err := url.Parse(uri)
	if err != nil {
		return "", "", err
	}

	if parsed.Scheme != "s3" {
		return "", "", errors.New("scheme must be 's3'")
	}

	bucket := parsed.Host
	if bucket == "" {
		return "", "", errors.New("bucket must not be empty")
	}

	var key string
	if parsed.Path != "" {
		key = parsed.Path[1:]
	}

	return bucket, key, nil
}

// makeURI constructs an S3 uri from the bucket and key to the format 's3://bucket/key'
func makeURI(bucket string, key string) string {
	return fmt.Sprintf("s3://%s/%s", bucket, key)

View on GitHub (pinned to 12126d8942)

Solutions

Use proper s3 URIs: s3://<bucket>/<key> (e.g. s3://my-bucket/data/file.json).
Strip or translate other schemes before calling: convert https://bucket/key to s3://bucket/key.
If you need http/gcs/local access, use the corresponding filesystem package instead of the s3 driver.
Validate the URI prefix (strings.HasPrefix(uri, "s3://")) in your configuration code before constructing the pipeline.

Example fix

// before
uri := "https://my-bucket.s3.amazonaws.com/data/file.json" // wrong scheme

// after
uri := "s3://my-bucket/data/file.json"

Defensive patterns

Strategy: validation

Validate before calling

u, err := url.Parse(uri)
if err != nil || u.Scheme != "s3" || u.Host == "" {
	return fmt.Errorf("expected s3://<bucket>/<key>, got %q", uri)
}

Prevention

Normalize all object paths to s3://bucket/key form before passing to the s3 filesystem.
Validate URI scheme and bucket in configuration-loading code.
Do not reuse https:// console URLs or gs:// / s3a:// URIs with the s3 driver.

When it happens

Trigger: Registering/using the s3 filesystem but passing a URI like 'https://bucket/key', 'gs://bucket/key', '/local/path/file', or 'S3://bucket/key' (uppercase scheme — url.Parse lowercases, but a different scheme string like 's3a' fails) to any of the s3 filesystem functions.

Common situations: Hardcoding an https console URL instead of the s3 object URI; copying pipeline code between filesystem drivers (gcs/http) without changing the scheme; using s3a:// or s3n:// URIs from Hadoop-style configs; environment-driven bucket paths that default to local paths.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8d49de27d3bf83b7. Report an issue: GitHub.