apache/beam · error · RuntimeError

AWS dependencies are not installed, and no alternative…

Error message

AWS dependencies are not installed, and no alternative client was provided to S3IO.

What it means

S3IO.__init__ raises RuntimeError when neither a pre-built client object nor the boto3 dependency is available. The class needs some AWS client to talk to S3; if boto3 is not installed in the environment and the caller did not inject an alternative client, there is no way to construct the IO wrapper.

Solutions

1. Install the AWS extra: pip install 'apache-beam[aws]' (or pip install boto3).
2. Pass an explicit client: S3IO(client=my_boto3_s3_client, options=options).
3. Verify installation with python -c "import boto3" in the exact interpreter/venv the pipeline runs in.
4. If boto3 truly cannot be installed, read/write via a different filesystem (e.g. copy out of S3 with the aws CLI first).

Example fix

# before
io = s3io.S3IO(options=options)  # RuntimeError: boto3 missing
# after
pip install 'apache-beam[aws]'
io = s3io.S3IO(client=boto3.client('s3'), options=options)

try:
    import boto3  # noqa: F401
except ImportError:
    raise RuntimeError("Install 'apache-beam[aws]' or pass client= to S3IO")

def can_use_s3io(client=None):
    return client is not None or BOTO3_INSTALLED

try:
    io = s3io.S3IO(options=options)
except RuntimeError as e:
    logging.error("S3IO unavailable: %s", e)
    io = None  # fall back to another filesystem

When it happens

Trigger: Instantiating S3IO() (or anything that does so, e.g. s3io.S3IO(options=...)) on a Python environment where apache-beam was installed without the gcp/aws extra and no `client=` argument is passed.

Common situations: Slim Docker images or CI runners without boto3; users who installed apache-beam bare (pip install apache-beam) instead of apache-beam[aws]; vendored environments where boto3 was stripped; passing options but forgetting client.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• Basepath %r must be S3 path.
• Cannot rename a directory
• ENOENT
• Invalid file open mode
• Invalid path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/be470c5644eea38d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3io.py:68

  match = re.match('^s3://([^/]+)/(.*)$', s3_path)
  if match is None or (match.group(2) == '' and not object_optional):
    raise ValueError('S3 path must be in the form s3://<bucket>/<object>.')
  return match.group(1), match.group(2)


class S3IO(object):
  """S3 I/O client."""
  def __init__(self, client=None, options=None):
    if client is None and options is None:
      raise ValueError('Must provide one of client or options')
    if client is not None:
      self.client = client
    elif BOTO3_INSTALLED:
      self.client = boto3_client.Client(options=options)
    else:
      message = 'AWS dependencies are not installed, and no alternative ' \
      'client was provided to S3IO.'
      raise RuntimeError(message)

  def open(
      self,
      filename,
      mode='r',
      read_buffer_size=16 * 1024 * 1024,
      mime_type='application/octet-stream'):
    """Open an S3 file path for reading or writing.

    Args:
      filename (str): S3 file path in the form ``s3://<bucket>/<object>``.
      mode (str): ``'r'`` for reading or ``'w'`` for writing.
      read_buffer_size (int): Buffer size to use during read operations.
      mime_type (str): Mime type to set for write operations.

    Returns:
      S3 file object.

View on GitHub (pinned to 12126d8942)