apache/beam · error

invalid glob pattern

Error message

invalid glob pattern %q: %w

What it means

List on the GCS filesystem compiles the object-name glob into a regex before listing candidates; if globToRegex fails (e.g. unclosed '[' character class), the syntax error is wrapped as 'invalid glob pattern %q: %w'. No GCS request is made — this is purely client-side pattern validation.

Solutions

1. Correct the glob syntax (close the '[' character class, remove invalid tokens).
2. Escape literal brackets in the pattern.
3. Validate the pattern client-side (filesystem.Match or a compile check) before List.
4. Log the exact pattern string passed in, since interpolation errors are common.

Example fix

// before
files, err := fs.List(ctx, "gs://b/data[202")
// after
files, err := fs.List(ctx, "gs://b/data[202[3]]") // or fix/escape the class

func validGCSGlob(p string) error {
	depth := 0
	for _, r := range p {
		if r == '[' { depth++ }
		if r == ']' { depth--; if depth < 0 { return errors.New("unmatched ]") } }
	}
	if depth != 0 { return errors.New("unclosed '['") }
	return nil
}

if _, err := fs.List(ctx, object); err != nil && strings.Contains(err.Error(), "invalid glob pattern") {
	return fmt.Errorf("fix pattern %q: %w", object, err)
}

When it happens

Trigger: Calling fs.List(ctx, glob) with a malformed glob such as an unclosed character class, or any other pattern globToRegex cannot translate.

Common situations: Dynamically assembled paths with truncated bracket expressions; users porting shell globs with shell-specific semantics; typos in pattern strings.

Related errors

• syntax error: unclosed
• empty chunk
• err
• expected 1 option, got
• invalid glob pattern

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/16009d977b946202. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/gcs/gcs.go:189

}

func (f *fs) Close() error {
	return f.client.Close()
}

func (f *fs) List(ctx context.Context, glob string) ([]string, error) {
	bucket, object, err := gcsx.ParseObject(glob)
	if err != nil {
		return nil, err
	}

	// Compile the glob pattern to a regex. We use a custom glob-to-regex
	// translation that treats / as a regular character (not a separator),
	// since GCS object names are flat. This also supports ** for recursive
	// matching, similar to the Java and Python SDKs.
	re, err := globToRegex(object)
	if err != nil {
		return nil, fmt.Errorf("invalid glob pattern %q: %w", object, err)
	}

	var candidates []string

	// We handle globs by list all candidates and matching them here.
	// For now, we assume * is the first matching character to make a
	// prefix listing and not list the entire bucket.
	prefix := fsx.GetPrefix(object)
	it := f.client.Bucket(bucket).UserProject(billingProject).Objects(ctx, &storage.Query{
		Prefix: prefix,
	})
	for {
		obj, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return nil, err

View on GitHub (pinned to 12126d8942)