apache/beam · error · BeamIOError

Checksum operation failed

Error message

Checksum operation failed

What it means

Raised by S3FileSystem.checksum when S3IO.checksum(path) raises; wrapped as BeamIOError keyed by path. Checksum only works for existing S3 objects; S3 provides ETag-based checksums, so non-file paths or missing objects fail.

Solutions

1. Ensure the path points to an object, not a prefix/directory
2. Verify the object exists and the caller has s3:GetObject permission
3. Catch BeamIOError and fall back to size+mtime comparison for change detection
4. Check credentials/region if all checksum calls fail

Example fix

// before
ck = fs.checksum(prefix + '/')  # not a file
// after
if not prefix.endswith('/'):
    ck = fs.checksum(prefix)
else:
    raise ValueError('checksum requires a file path')

if path.endswith('/'):
    raise ValueError('checksum requires a file path, not an s3 prefix')

try:
    ck = fs.checksum(path)
except BeamIOError as e:
    log.error('checksum failed for %s: %s', path, e.exception_details)
    ck = None

When it happens

Trigger: Calling fs.checksum('s3://bucket/dir/') on a directory/prefix rather than a file; nonexistent object; boto3 failure during the underlying request.

Common situations: Passing an S3 prefix (folder) instead of an object key; comparing checksums of objects that were replaced during transfer; missing read permissions.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Basepath %r must be S3 path.
• Delete operation failed
• ENOENT
• error listing object keys
• exists() operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/979730191f2dbdfb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:283

    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("last_updated operation failed", {path: e})

  def checksum(self, path):
    """Fetch checksum metadata of a file on the
    :class:`~apache_beam.io.filesystem.FileSystem`.

    Args:
      path: string path of a file.

    Returns: string containing checksum

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).checksum(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Checksum operation failed", {path: e})

  def metadata(self, path):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      path: string path of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      file_metadata = s3io.S3IO(options=self._options)._status(path)
      return FileMetadata(
          path, file_metadata['size'], file_metadata['last_updated'])
    except Exception as e:  # pylint: disable=broad-except

View on GitHub (pinned to 12126d8942)