{"record":{"id":"287de8a2cc2b5b6e","repo":"apache/beam","slug":"path-r-must-be-s3-path","errorCode":null,"errorMessage":"Path %r must be S3 path.","messagePattern":"Path %r must be S3 path\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3filesystem.py","lineNumber":87,"sourceCode":"      path = path.rstrip('/') + '/' + p.lstrip('/')\n    return path\n\n  def split(self, path):\n    \"\"\"Splits the given path into two parts.\n\n    Splits the path into a pair (head, tail) such that tail contains the last\n    component of the path and head contains everything up to that.\n\n    Head will include the S3 prefix ('s3://').\n\n    Args:\n      path: path as a string\n    Returns:\n      a pair of path components as strings.\n    \"\"\"\n    path = path.strip()\n    if not path.startswith(S3FileSystem.S3_PREFIX):\n      raise ValueError('Path %r must be S3 path.' % path)\n\n    prefix_len = len(S3FileSystem.S3_PREFIX)\n    last_sep = path[prefix_len:].rfind('/')\n    if last_sep >= 0:\n      last_sep += prefix_len\n\n    if last_sep > 0:\n      return (path[:last_sep], path[last_sep + 1:])\n    elif last_sep < 0:\n      return (path, '')\n    else:\n      raise ValueError('Invalid path: %s' % path)\n\n  def mkdirs(self, path):\n    \"\"\"Recursively create directories for the provided path.\n\n    Args:\n      path: string path of the directory structure that should be created","sourceCodeStart":69,"sourceCodeEnd":105,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3filesystem.py#L69-L105","documentation":"S3FileSystem.split() raises ValueError('Path %r must be S3 path.') when the given path, after stripping whitespace, does not start with the s3:// prefix. Like join(), split() is S3-specific and refuses non-S3 URIs before computing (dirname, basename).","triggerScenarios":"Calling S3FileSystem().split('/local/file.txt'), split('bucket/file'), or a path with a malformed prefix ('s3:/x', 'S3://x' case-sensitivity) — directly or via code that splits result paths from copy/match operations.","commonSituations":"Passing local staging paths into S3-specific helpers; processing matched file paths whose scheme is not s3; pipeline sinks emitting paths in another scheme mixed with S3 code paths; accidental prefix stripping with lstrip or replace.","solutions":["Pass a full S3 URI beginning with 's3://' to split().","Check path.startswith(S3FileSystem.S3_PREFIX) before splitting and route non-S3 paths to their own filesystem.","Parse the scheme generically (urlparse / Beam's FileSystems) and dispatch per scheme instead of hardcoding S3FileSystem.","Watch for accidental prefix removal — avoid path.lstrip('s3:/') style transformations."],"exampleFix":"// before\nS3FileSystem().split('/tmp/prefix/file.avro')\n// after\nS3FileSystem().split('s3://bucket/prefix/file.avro')","handlingStrategy":"type-guard","validationCode":"def is_s3_path(p: str) -> bool:\n    return isinstance(p, str) and p.strip().startswith('s3://')","typeGuard":"def is_s3_path(p: str) -> bool:\n    return isinstance(p, str) and p.strip().startswith('s3://')","tryCatchPattern":"try:\n    dirpath, filename = S3FileSystem().split(path)\nexcept ValueError as e:\n    if 'must be S3 path' in str(e):\n        # dispatch to the filesystem matching the actual scheme\n        dirpath, filename = FileSystems.split(path)\n    else:\n        raise","preventionTips":["Verify path scheme before calling S3-specific helpers","Use FileSystems.split/join for scheme-agnostic path handling","Don't mix local staging paths with S3 paths in the same code path"],"tags":["s3","path","aws","io","python"],"backgroundTag":"invalid-url-format","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}