apache/beam · error · BeamIOError

Checksum operation failed

Error message

Checksum operation failed

What it means

GcsFileSystem.checksum wraps gcsio.checksum and raises BeamIOError('Checksum operation failed') whenever the underlying call throws, e.g. because the path is not a file or does not exist. The original error is stored in exception_details keyed by path.

Solutions

1. Verify the path exists and is a file (filesystem.match/exists) before calling checksum.
2. Check the wrapped error in exception_details for the root cause.
3. Correct the gs:// path (bucket + full object name).
4. Catch BeamIOError and treat missing objects as empty/absent checksums if appropriate.

Example fix

// before
crc = fs.checksum(path)
// after
if fs.exists(path):
  crc = fs.checksum(path)
else:
  crc = None

if not fs.exists(path):
    raise FileNotFoundError(path)

def valid_gcs_path(p):
    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3

try:
    crc = fs.checksum(path)
except BeamIOError as e:
    logging.warning('checksum failed: %s', e.exception_details.get(path))
    crc = None

When it happens

Trigger: Calling GcsFileSystem.checksum(path) on a path that doesn't exist, is a directory/prefix, or where the GCS API call fails (permissions, network).

Common situations: Computing checksums of output files deleted by retention policies; passing a directory prefix instead of a single object path; typo'd object name.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Checksum operation failed
• Copy operation failed
• Delete operation failed
• dynamic: wrapped StorageException via…
• empty chunk

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/02e41040db876327. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:325

    """
    return self._gcsIO().last_updated(path)

  def checksum(self, path):
    """Fetch checksum metadata of a file on the
    :class:`~apache_beam.io.filesystem.FileSystem`.

    Args:
      path: string path of a file.

    Returns: string containing checksum

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      return self._gcsIO().checksum(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Checksum operation failed", {path: e})

  def metadata(self, path):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      path: string path of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      file_metadata = self._gcsIO()._status(path)
      return FileMetadata(path, file_metadata['size'], file_metadata['updated'])
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Metadata operation failed", {path: e})

View on GitHub (pinned to 12126d8942)