apache/beam · error · ValueError

Basepath %r must be S3 path.

Error message

Basepath %r must be S3 path.

What it means

S3FileSystem.join() raises ValueError('Basepath %r must be S3 path.') when the first path component does not start with the S3 prefix (s3://). The method only knows how to build S3 URLs, so a basepath like a local path or bare bucket name is rejected before joining.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:65

    self._options = pipeline_options

  @classmethod
  def scheme(cls):
    """URI scheme for the FileSystem
    """
    return 's3'

  def join(self, basepath, *paths):
    """Join two or more pathname components for the filesystem

    Args:
      basepath: string path of the first component of the path
      paths: path components to be added

    Returns: full path after combining all of the return nulled components
    """
    if not basepath.startswith(S3FileSystem.S3_PREFIX):
      raise ValueError('Basepath %r must be S3 path.' % basepath)

    path = basepath
    for p in paths:
      path = path.rstrip('/') + '/' + p.lstrip('/')
    return path

  def split(self, path):
    """Splits the given path into two parts.

    Splits the path into a pair (head, tail) such that tail contains the last
    component of the path and head contains everything up to that.

    Head will include the S3 prefix ('s3://').

    Args:
      path: path as a string
    Returns:
      a pair of path components as strings.

View on GitHub (pinned to 12126d8942)

Solutions

  1. Ensure the basepath starts with S3FileSystem.S3_PREFIX ('s3://') before calling join.
  2. Prepend the prefix: S3FileSystem.S3_PREFIX + path if you know the path is an S3 reference.
  3. Normalize with match_basepath/strip_prefix helpers or validate the URL with FileSystemURI helpers first.
  4. Use a generic filesystem (FileSystems.get_scheme-based) so join dispatches to the right filesystem.

Example fix

// before
S3FileSystem().join('bucket/output', 'part-0.avro')
// after
S3FileSystem().join('s3://bucket/output', 'part-0.avro')
Defensive patterns

Strategy: validation

Validate before calling

def ensure_s3_path(basepath: str) -> str:
    return basepath if basepath.startswith('s3://') else 's3://' + basepath.lstrip('/')

Type guard

def is_s3_path(p: str) -> bool:
    return isinstance(p, str) and p.startswith('s3://')

Try / catch

try:
    full = S3FileSystem().join(basepath, *parts)
except ValueError as e:
    if 'must be S3 path' in str(e):
        full = S3FileSystem().join('s3://' + basepath.lstrip('/'), *parts)
    else:
        raise

Prevention

When it happens

Trigger: Calling S3FileSystem().join(basepath, *paths) where basepath lacks the 's3://' prefix — e.g. passing '/tmp/data', 'bucket/key', or a GCS path.

Common situations: Mixing local and S3 paths in file staging code; generic filesystem helpers passing any path into a hardcoded S3FileSystem; typos like 's3:/bucket' or stripped prefixes; tests that forget to prefix fixture paths.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f9074e1ed7241bbf. Report an issue: GitHub.