apache/beam · error

object must have 'gs' scheme

Error message

object %s must have 'gs' scheme

What it means

gcsx.ParseObject parses a GCS object URL and requires the 'gs' scheme. This error is returned when the given object string parses as a URL but its scheme is not 'gs' (e.g. 's3://...', 'https://...', or a bare path). Callers like List, OpenRead, staging/retrieval servers reject the path before touching GCS.

Solutions

1. Use the full gs://bucket/object URI, e.g. 'gs://my-bucket/path/obj'
2. If the source is another provider, migrate/download the object to GCS first
3. Validate the scheme in your config before calling the library

Example fix

// before
gcsx.OpenRead(ctx, client, "https://storage.googleapis.com/my-bucket/obj")
// after
gcsx.OpenRead(ctx, client, "gs://my-bucket/obj")

u, err := url.Parse(obj)
if err != nil || u.Scheme != "gs" {
	return fmt.Errorf("need gs:// URI, got %q", obj)
}

func isGCSURI(s string) bool {
	u, err := url.Parse(s)
	return err == nil && u.Scheme == "gs"
}

if err != nil {
	if strings.Contains(err.Error(), "must have 'gs' scheme") {
		return fmt.Errorf("invalid GCS path %q: prefix with gs://", obj)
	}
	return err
}

When it happens

Trigger: Passing an object string with a non-gs scheme (http://, s3://, /local/path) to gcsx.List, gcsx.OpenRead, ParseObject, NewStagingServer/NewRetrievalServer manifest paths, or ReadProxyManifest.

Common situations: Config mistakes: pointing staging/manifest options at local paths or other cloud URLs, forgetting the gs:// prefix, or copy-pasting an HTTPS console URL instead of the gs:// URI.

Understand the failure class

Background: "Invalid URL" / "URL cannot be empty": fix the malformed or missing URL behind request-construction failures — this error's family across 50 libraries.

Related errors

• object must have bucket
• invalid manifest location
• location is not a GCS object
• project cannot be empty, got
• A VPC network must be provided to use a private endpoint.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7b96a8bab7bbaa24. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/util/gcsx/gcs.go:142

// MakeObject creates a object location from bucket and path. For example,
// MakeObject("foo", "bar/baz") returns "gs://foo/bar/baz". The bucket
// must be non-empty.
func MakeObject(bucket, path string) string {
	if bucket == "" {
		panic("bucket must be non-empty")
	}
	return fmt.Sprintf("gs://%v/%v", bucket, path)
}

// ParseObject deconstructs a GCS object name into (bucket, name).
func ParseObject(object string) (bucket, path string, err error) {
	parsed, err := url.Parse(object)
	if err != nil {
		return "", "", err
	}

	if parsed.Scheme != "gs" {
		return "", "", errors.Errorf("object %s must have 'gs' scheme", object)
	}
	if parsed.Host == "" {
		return "", "", errors.Errorf("object %s must have bucket", object)
	}
	if parsed.Path == "" {
		return parsed.Host, "", nil
	}

	// remove leading "/" in URL path
	return parsed.Host, parsed.Path[1:], nil
}

// Join joins a GCS path with an element. Preserves
// the gs:// prefix.
func Join(object string, elms ...string) string {
	bucket, prefix, err := ParseObject(object)
	if err != nil {
		panic(err)

View on GitHub (pinned to 12126d8942)