apache/beam · error · ValueError

Invalid path: %s

Error message

Invalid path: %s

What it means

S3FileSystem.split() raises ValueError('Invalid path: %s') when the path starts with s3:// but has no '/' after the prefix at all (last_sep == 0 position case), i.e. something like 's3://' with no bucket/separator to split on. It's the degenerate-path guard after prefix and separator analysis.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:99

      path: path as a string
    Returns:
      a pair of path components as strings.
    """
    path = path.strip()
    if not path.startswith(S3FileSystem.S3_PREFIX):
      raise ValueError('Path %r must be S3 path.' % path)

    prefix_len = len(S3FileSystem.S3_PREFIX)
    last_sep = path[prefix_len:].rfind('/')
    if last_sep >= 0:
      last_sep += prefix_len

    if last_sep > 0:
      return (path[:last_sep], path[last_sep + 1:])
    elif last_sep < 0:
      return (path, '')
    else:
      raise ValueError('Invalid path: %s' % path)

  def mkdirs(self, path):
    """Recursively create directories for the provided path.

    Args:
      path: string path of the directory structure that should be created

    Raises:
      IOError: if leaf directory already exists.
    """
    pass

  def has_dirs(self):
    """Whether this FileSystem supports directories."""
    return False

  def _list(self, dir_or_prefix):
    """List files in a location.

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass a complete S3 URI of the form s3://bucket/path before splitting.
2. Validate that len(path) > len('s3://') and contains a bucket component before calling split.
3. Fix path-building code that concatenates scheme and suffix (missing bucket between).
4. Guard upstream config so empty base paths are rejected before reaching filesystem calls.

Example fix

// before
S3FileSystem().split('s3://')
// after
S3FileSystem().split('s3://bucket/key')

def is_complete_s3_uri(p: str) -> bool:
    return isinstance(p, str) and p.startswith('s3://') and len(p) > len('s3://') and '/' in p[len('s3://'):]

try:
    dirpath, filename = S3FileSystem().split(path)
except ValueError as e:
    if str(e).startswith('Invalid path'):
        raise ValueError(f"S3 path needs bucket and separator: {path!r}")
    raise

When it happens

Trigger: Calling S3FileSystem().split('s3://') or a path where rfind('/') over path[prefix_len:] returns -1 in combination with an edge position producing last_sep == 0 — paths with nothing beyond the scheme.

Common situations: Empty or placeholder S3 URIs coming from unset config variables; constructing paths by string concatenation where the bucket part was lost; stripping components down to the bare scheme.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Basepath %r must be S3 path.
• Path %r must be S3 path.
• ENOENT
• List operation failed
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5c11346c97098627. Report an issue: GitHub.