apache/beam · critical · RuntimeError

Could not find python-snappy, google-crc32c, or crcmod. To a

Error message

Could not find python-snappy, google-crc32c, or crcmod. To allow execution to succeed, make sure that one of these packages is installed or pip install apache-beam[tfrecord]

What it means

_default_crc32c_fn needs a CRC32C implementation (python-snappy, google-crc32c, or crcmod) to compute TFRecord checksums. If none is importable and crcmod is None, it raises RuntimeError because TFRecord reading requires a CRC function.

Source

Thrown at sdks/python/apache_beam/io/tfrecordio.py:74

      if getattr(snappy, '_crc32c', None):
        _default_crc32c_fn.fn = snappy._crc32c  # pylint: disable=protected-access
      elif getattr(snappy, '_snappy', None):
        _default_crc32c_fn.fn = snappy._snappy._crc32c  # pylint: disable=protected-access
    except ImportError:
      pass

  if not _default_crc32c_fn.fn:
    try:
      import google_crc32c  # pylint: disable=import-error

      if getattr(google_crc32c, 'value', None):
        _default_crc32c_fn.fn = google_crc32c.value  # pylint: disable=protected-access
    except ImportError:
      pass

    if not _default_crc32c_fn.fn:
      if crcmod is None:
        raise RuntimeError(
            'Could not find python-snappy, google-crc32c, or crcmod. To allow '
            'execution to succeed, make sure that one of these packages is '
            'installed or pip install apache-beam[tfrecord]')
      _LOGGER.warning(
          'Couldn\'t find python-snappy or google-crc32c so the '
          'implementation of _TFRecordUtil._masked_crc32c is not as fast '
          'as it could be.')
      _default_crc32c_fn.fn = crcmod.predefined.mkPredefinedCrcFun('crc-32c')
  return _default_crc32c_fn.fn(value)


_default_crc32c_fn.fn = None  # type: ignore


class _TFRecordUtil(object):
  """Provides basic TFRecord encoding/decoding with consistency checks.

  For detailed TFRecord format description see:

View on GitHub (pinned to 12126d8942)

Solutions

pip install apache-beam[tfrecord]
Or install one package directly: pip install crcmod, google-crc32c, or python-snappy
Add the package to requirements.txt so remote workers get it

Example fix

// before
# no crc32c provider installed
// after
pip install apache-beam[tfrecord]

Defensive patterns

Strategy: fallback

Validate before calling

try:
    import crcmod  # or google_crc32c / snappy
    CRC_OK = True
except ImportError:
    CRC_OK = False

Try / catch

try:
    data = beam.io.ReadFromTFRecord(path)
except RuntimeError as e:
    if 'python-snappy' in str(e):
        raise SystemExit('Install apache-beam[tfrecord] before running')
    raise

Prevention

Use the apache-beam[tfrecord] extra when installing
Add crcmod or google-crc32c to requirements.txt for remote runners
Verify extras in Docker images used for workers

When it happens

Trigger: Reading TFRecord files (_TFRecordUtil._masked_crc32c) in an environment lacking all three packages.

Common situations: Installing apache-beam without the [tfrecord] extra; slim runtime images on Dataflow/Flink runners.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4386bcc5e55d1cf1. Report an issue: GitHub.