apache/beam · error · ValueError

Must provide one of client or options

Error message

Must provide one of client or options

What it means

S3IO's constructor requires either a pre-configured boto3 client or an options object to build one; passing neither leaves the client with no credentials/endpoint configuration, so the guard rejects the ambiguous initialization.

Solutions

1. Pass options=PipelineOptions(['--s3_access_key', ..., '--s3_secret_key', ...]) or equivalent S3 options
2. Construct a boto3 client yourself and pass client=client
3. When using S3FileSystem, ensure it is initialized with non-None pipeline options
4. If boto3 is missing, install apache-beam with the aws extra: pip install apache-beam[gcp,aws]

Example fix

// before
io_client = s3io.S3IO()
// after
io_client = s3io.S3IO(options=PipelineOptions(['--s3_endpoint', 'https://s3.amazonaws.com']))

def make_s3io(client=None, options=None):
    if client is None and options is None:
        raise ValueError('S3IO requires client or options')
    return s3io.S3IO(client=client, options=options)

try:
    io = s3io.S3IO(options=opts)
except ValueError as e:
    log.error('S3IO construction failed: %s', e)
    io = s3io.S3IO(client=default_boto3_client())

When it happens

Trigger: Instantiating s3io.S3IO() with no arguments, or S3FileSystem paths where self._options is None; calling S3IO directly in scripts without options.

Common situations: Quick scripts calling S3IO directly instead of through S3FileSystem; a PipelineOptions that lost AWS options; test code constructing S3IO without fixtures.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• S3 path must be in the form s3://
• Tried to list nonexistent S3 path: s3://
• All parts but the last must be larger than
• AWS dependencies are not installed, and no alternative…
• Basepath %r must be S3 path.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5512235820f359b3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3io.py:60

except ImportError:
  BOTO3_INSTALLED = False

MAX_BATCH_OPERATION_SIZE = 100


def parse_s3_path(s3_path, object_optional=False):
  """Return the bucket and object names of the given s3:// path."""
  match = re.match('^s3://([^/]+)/(.*)$', s3_path)
  if match is None or (match.group(2) == '' and not object_optional):
    raise ValueError('S3 path must be in the form s3://<bucket>/<object>.')
  return match.group(1), match.group(2)


class S3IO(object):
  """S3 I/O client."""
  def __init__(self, client=None, options=None):
    if client is None and options is None:
      raise ValueError('Must provide one of client or options')
    if client is not None:
      self.client = client
    elif BOTO3_INSTALLED:
      self.client = boto3_client.Client(options=options)
    else:
      message = 'AWS dependencies are not installed, and no alternative ' \
      'client was provided to S3IO.'
      raise RuntimeError(message)

  def open(
      self,
      filename,
      mode='r',
      read_buffer_size=16 * 1024 * 1024,
      mime_type='application/octet-stream'):
    """Open an S3 file path for reading or writing.

    Args:

View on GitHub (pinned to 12126d8942)