apache/beam · error · ValueError

Invalid file open mode

Error message

Invalid file open mode: %s.

What it means

S3IO.open only accepts modes 'r'/'rb' (read) and 'w'/'wb' (write); any other mode string raises ValueError. S3 objects do not support append or read-write modes, so the wrapper deliberately rejects them.

Solutions

1. Use only 'r', 'rb', 'w', or 'wb' as the mode.
2. For append semantics, read the whole object, rewrite with mode='w' (S3 has no append).
3. If a generic API passes modes through, normalize/whitelist the mode before calling open().

Example fix

// before
f = s3io.open('s3://bucket/log', mode='a')  # ValueError
// after
f = s3io.open('s3://bucket/log', mode='wb')  # full rewrite; S3 cannot append

VALID_S3_MODES = {'r', 'rb', 'w', 'wb'}
if mode not in VALID_S3_MODES:
    raise ValueError(f"S3IO supports only {VALID_S3_MODES}, got {mode!r}")

def is_valid_s3_mode(mode):
    return mode in ('r', 'rb', 'w', 'wb')

try:
    f = io.open(path, mode=mode)
except ValueError as e:
    logging.error("bad mode %r: %s", mode, e)
    f = io.open(path, mode='rb')  # sane default

When it happens

Trigger: Calling S3IO.open('s3://bucket/key', mode='a'), mode='r+', mode='x', or passing a mode with encoding suffixes like 'rt'.

Common situations: Porting local-file code that appends logs to S3; generic file helper functions that take a mode parameter and pass it through; using text mode flags like 'r+t' assumed to be normalized.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• AWS dependencies are not installed, and no alternative…
• Basepath %r must be S3 path.
• Cannot rename a directory
• ENOENT
• Invalid path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7312ce12081fd04d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3io.py:100

      mime_type (str): Mime type to set for write operations.

    Returns:
      S3 file object.

    Raises:
      ValueError: Invalid open file mode.
    """
    if mode == 'r' or mode == 'rb':
      downloader = S3Downloader(
          self.client, filename, buffer_size=read_buffer_size)
      return io.BufferedReader(
          DownloaderStream(downloader, mode=mode), buffer_size=read_buffer_size)
    elif mode == 'w' or mode == 'wb':
      uploader = S3Uploader(self.client, filename, mime_type)
      return io.BufferedWriter(
          UploaderStream(uploader, mode=mode), buffer_size=128 * 1024)
    else:
      raise ValueError('Invalid file open mode: %s.' % mode)

  def list_files(self, path, with_metadata=False):
    """Lists files matching the prefix.

    Args:
      path: S3 file path pattern in the form s3://<bucket>/[name].
      with_metadata: Experimental. Specify whether returns file metadata.

    Returns:
      If ``with_metadata`` is False: generator of tuple(file name, size); if
      ``with_metadata`` is True: generator of
      tuple(file name, tuple(size, timestamp)).
    """
    bucket, prefix = parse_s3_path(path, object_optional=True)
    request = messages.ListRequest(bucket=bucket, prefix=prefix)

    file_info = set()
    counter = 0

View on GitHub (pinned to 12126d8942)