apache/beam · error

invalid key pattern

Error message

invalid key pattern: %s

What it means

This is returned when filepath.Match fails to compile the caller-supplied glob keyPattern against a key. filepath.Match returns ErrBadPattern for malformed patterns like an unterminated '[' escape. Note the actual pattern (not the underlying error) is reported, which can obscure the root cause.

Solutions

1. Fix the glob pattern syntax: balance all '[' ... ']' character classes.
2. Remember filepath.Match has no '**' — use '*' per path segment.
3. Pre-validate the pattern with filepath.Match(pattern, "") and check for filepath.ErrBadPattern before calling List.
4. If the pattern comes from user config, add validation at config load time.

Example fix

// before
keys, err := fs.List(ctx, "s3://bucket/data[abc")
// after
if _, err := filepath.Match("data[abc", "probe"); err != nil {
  return fmt.Errorf("bad glob: %w", err)
}
keys, err := fs.List(ctx, "s3://bucket/data[abc]")

func validGlob(pattern string) bool {
  _, err := filepath.Match(pattern, "probe")
  return err == nil // false when pattern has filepath.ErrBadPattern
}

match, err := filepath.Match(keyPattern, key)
if errors.Is(err, filepath.ErrBadPattern) {
  return fmt.Errorf("glob %q is malformed (check [ ] classes): %w", keyPattern, err)
}

When it happens

Trigger: Passing a glob with bad syntax to List(), e.g. 's3://bucket/data[abc' (unclosed character class) or a trailing lone backslash; the error fires on the first object key checked.

Common situations: Users constructing patterns by string concatenation accidentally producing unbalanced brackets; porting shell globs with syntax filepath.Match doesn't support (e.g. '**').

Related errors

• error parsing S3 destination uri
• error parsing S3 source uri
• error parsing S3 uri
• error parsing S3 uri
• invalid glob pattern

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0597e5f206367c78. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:105

	prefix := fsx.GetPrefix(keyPattern)
	params := &s3.ListObjectsV2Input{
		Bucket: aws.String(bucket),
		Prefix: aws.String(prefix),
	}
	paginator := s3.NewListObjectsV2Paginator(f.client, params)

	var objects []string
	for paginator.HasMorePages() {
		output, err := paginator.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("error retrieving page: %v", err)
		}

		for _, object := range output.Contents {
			key := aws.ToString(object.Key)
			match, err := filepath.Match(keyPattern, key)
			if err != nil {
				return nil, fmt.Errorf("invalid key pattern: %s", keyPattern)
			}

			if match {
				objects = append(objects, key)
			}
		}
	}

	return objects, nil
}

// OpenRead returns a new io.ReadCloser to read contents from the file. The caller must call Close
// on the returned io.ReadCloser when done reading.
func (f *fs) OpenRead(ctx context.Context, filename string) (io.ReadCloser, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

View on GitHub (pinned to 12126d8942)