apache/beam · error · ValueError

S3 path must be in the form s3://

Error message

S3 path must be in the form s3://<bucket>/<object>.

What it means

parse_s3_path validates that a path matches s3://<bucket>/<object> using a regex and raises ValueError otherwise. With object_optional=True an object part may be empty (bucket-only path), but the s3:// scheme and non-empty bucket are always required.

Solutions

1. Ensure the path starts with s3:// and has both bucket and object: s3://my-bucket/my/key
2. Use s3io.parse_s3_path(path, object_optional=True) when a bucket-root path is legitimate
3. Fix the scheme if a gs:// or bare path was passed to the S3 filesystem
4. Add a regex/parse check on user-supplied paths before passing them into S3IO APIs

Example fix

// before
s3io.S3IO(options=options).copy_files(['bucket/key'], ['other-bucket/key'])
// after
paths = ['s3://bucket/key']
dests = ['s3://other-bucket/key']
s3io.S3IO(options=options).copy_files(paths, dests)

import re
def is_valid_s3_path(p, object_optional=False):
    m = re.match(r'^s3://([^/]+)/(.*)$', p)
    return m is not None and (m.group(2) != '' or object_optional)

try:
    bucket, obj = s3io.parse_s3_path(path)
except ValueError as e:
    log.error('bad s3 path %r: %s', path, e)
    return None

When it happens

Trigger: Passing paths like 's3://bucket' (no trailing slash/object, object_optional=False), 's3:///key' (empty bucket), 'gs://bucket/key', plain 'bucket/key', or URLs with typos (e.g. 's3:/bucket/key').

Common situations: Users mixing GCS-style paths into S3 connectors; constructing paths via string concatenation and dropping a slash; passing local filesystem paths to S3IO; config values pointing at the wrong scheme.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Must provide one of client or options
• All parts but the last must be larger than
• AWS dependencies are not installed, and no alternative…
• Basepath %r must be GCS path.
• Basepath %r must be S3 path.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/128d4f7fd5ce893e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3io.py:52

from apache_beam.io.filesystemio import UploaderStream
from apache_beam.utils import retry

try:
  # pylint: disable=wrong-import-order, wrong-import-position
  # pylint: disable=ungrouped-imports
  from apache_beam.io.aws.clients.s3 import boto3_client
  BOTO3_INSTALLED = True
except ImportError:
  BOTO3_INSTALLED = False

MAX_BATCH_OPERATION_SIZE = 100


def parse_s3_path(s3_path, object_optional=False):
  """Return the bucket and object names of the given s3:// path."""
  match = re.match('^s3://([^/]+)/(.*)$', s3_path)
  if match is None or (match.group(2) == '' and not object_optional):
    raise ValueError('S3 path must be in the form s3://<bucket>/<object>.')
  return match.group(1), match.group(2)


class S3IO(object):
  """S3 I/O client."""
  def __init__(self, client=None, options=None):
    if client is None and options is None:
      raise ValueError('Must provide one of client or options')
    if client is not None:
      self.client = client
    elif BOTO3_INSTALLED:
      self.client = boto3_client.Client(options=options)
    else:
      message = 'AWS dependencies are not installed, and no alternative ' \
      'client was provided to S3IO.'
      raise RuntimeError(message)

  def open(

View on GitHub (pinned to 12126d8942)