{"record":{"id":"128d4f7fd5ce893e","repo":"apache/beam","slug":"s3-path-must-be-in-the-form-s3-bucket-object","errorCode":null,"errorMessage":"S3 path must be in the form s3://<bucket>/<object>.","messagePattern":"S3 path must be in the form s3://<bucket>/<object>\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3io.py","lineNumber":52,"sourceCode":"from apache_beam.io.filesystemio import UploaderStream\nfrom apache_beam.utils import retry\n\ntry:\n  # pylint: disable=wrong-import-order, wrong-import-position\n  # pylint: disable=ungrouped-imports\n  from apache_beam.io.aws.clients.s3 import boto3_client\n  BOTO3_INSTALLED = True\nexcept ImportError:\n  BOTO3_INSTALLED = False\n\nMAX_BATCH_OPERATION_SIZE = 100\n\n\ndef parse_s3_path(s3_path, object_optional=False):\n  \"\"\"Return the bucket and object names of the given s3:// path.\"\"\"\n  match = re.match('^s3://([^/]+)/(.*)$', s3_path)\n  if match is None or (match.group(2) == '' and not object_optional):\n    raise ValueError('S3 path must be in the form s3://<bucket>/<object>.')\n  return match.group(1), match.group(2)\n\n\nclass S3IO(object):\n  \"\"\"S3 I/O client.\"\"\"\n  def __init__(self, client=None, options=None):\n    if client is None and options is None:\n      raise ValueError('Must provide one of client or options')\n    if client is not None:\n      self.client = client\n    elif BOTO3_INSTALLED:\n      self.client = boto3_client.Client(options=options)\n    else:\n      message = 'AWS dependencies are not installed, and no alternative ' \\\n      'client was provided to S3IO.'\n      raise RuntimeError(message)\n\n  def open(","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3io.py#L34-L70","documentation":"parse_s3_path validates that a path matches s3://<bucket>/<object> using a regex and raises ValueError otherwise. With object_optional=True an object part may be empty (bucket-only path), but the s3:// scheme and non-empty bucket are always required.","triggerScenarios":"Passing paths like 's3://bucket' (no trailing slash/object, object_optional=False), 's3:///key' (empty bucket), 'gs://bucket/key', plain 'bucket/key', or URLs with typos (e.g. 's3:/bucket/key').","commonSituations":"Users mixing GCS-style paths into S3 connectors; constructing paths via string concatenation and dropping a slash; passing local filesystem paths to S3IO; config values pointing at the wrong scheme.","solutions":["Ensure the path starts with s3:// and has both bucket and object: s3://my-bucket/my/key","Use s3io.parse_s3_path(path, object_optional=True) when a bucket-root path is legitimate","Fix the scheme if a gs:// or bare path was passed to the S3 filesystem","Add a regex/parse check on user-supplied paths before passing them into S3IO APIs"],"exampleFix":"// before\ns3io.S3IO(options=options).copy_files(['bucket/key'], ['other-bucket/key'])\n// after\npaths = ['s3://bucket/key']\ndests = ['s3://other-bucket/key']\ns3io.S3IO(options=options).copy_files(paths, dests)","handlingStrategy":"validation","validationCode":"import re\ndef is_valid_s3_path(p, object_optional=False):\n    m = re.match(r'^s3://([^/]+)/(.*)$', p)\n    return m is not None and (m.group(2) != '' or object_optional)","typeGuard":null,"tryCatchPattern":"try:\n    bucket, obj = s3io.parse_s3_path(path)\nexcept ValueError as e:\n    log.error('bad s3 path %r: %s', path, e)\n    return None","preventionTips":["Normalize all paths to s3://<bucket>/<object> at ingestion boundaries","Use object_optional=True only when bucket-root paths are intended","Sanitize gs:// or local paths before routing to the S3 filesystem"],"tags":["s3","path-validation","aws","valueerror"],"backgroundTag":"invalid-url-format","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}