apache/beam · error · ValueError

Path %r must be S3 path.

Error message

Path %r must be S3 path.

What it means

S3FileSystem.split() raises ValueError('Path %r must be S3 path.') when the given path, after stripping whitespace, does not start with the s3:// prefix. Like join(), split() is S3-specific and refuses non-S3 URIs before computing (dirname, basename).

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:87

      path = path.rstrip('/') + '/' + p.lstrip('/')
    return path

  def split(self, path):
    """Splits the given path into two parts.

    Splits the path into a pair (head, tail) such that tail contains the last
    component of the path and head contains everything up to that.

    Head will include the S3 prefix ('s3://').

    Args:
      path: path as a string
    Returns:
      a pair of path components as strings.
    """
    path = path.strip()
    if not path.startswith(S3FileSystem.S3_PREFIX):
      raise ValueError('Path %r must be S3 path.' % path)

    prefix_len = len(S3FileSystem.S3_PREFIX)
    last_sep = path[prefix_len:].rfind('/')
    if last_sep >= 0:
      last_sep += prefix_len

    if last_sep > 0:
      return (path[:last_sep], path[last_sep + 1:])
    elif last_sep < 0:
      return (path, '')
    else:
      raise ValueError('Invalid path: %s' % path)

  def mkdirs(self, path):
    """Recursively create directories for the provided path.

    Args:
      path: string path of the directory structure that should be created

View on GitHub (pinned to 12126d8942)

Solutions

  1. Pass a full S3 URI beginning with 's3://' to split().
  2. Check path.startswith(S3FileSystem.S3_PREFIX) before splitting and route non-S3 paths to their own filesystem.
  3. Parse the scheme generically (urlparse / Beam's FileSystems) and dispatch per scheme instead of hardcoding S3FileSystem.
  4. Watch for accidental prefix removal — avoid path.lstrip('s3:/') style transformations.

Example fix

// before
S3FileSystem().split('/tmp/prefix/file.avro')
// after
S3FileSystem().split('s3://bucket/prefix/file.avro')
Defensive patterns

Strategy: type-guard

Validate before calling

def is_s3_path(p: str) -> bool:
    return isinstance(p, str) and p.strip().startswith('s3://')

Type guard

def is_s3_path(p: str) -> bool:
    return isinstance(p, str) and p.strip().startswith('s3://')

Try / catch

try:
    dirpath, filename = S3FileSystem().split(path)
except ValueError as e:
    if 'must be S3 path' in str(e):
        # dispatch to the filesystem matching the actual scheme
        dirpath, filename = FileSystems.split(path)
    else:
        raise

Prevention

When it happens

Trigger: Calling S3FileSystem().split('/local/file.txt'), split('bucket/file'), or a path with a malformed prefix ('s3:/x', 'S3://x' case-sensitivity) — directly or via code that splits result paths from copy/match operations.

Common situations: Passing local staging paths into S3-specific helpers; processing matched file paths whose scheme is not s3; pipeline sinks emitting paths in another scheme mixed with S3 code paths; accidental prefix stripping with lstrip or replace.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/287de8a2cc2b5b6e. Report an issue: GitHub.