apache/beam · error · ValueError

Not a valid TFRecord. Mismatch of length mask: %s

Error message

Not a valid TFRecord. Mismatch of length mask: %s

What it means

After unpacking the length header, the reader recomputes the masked CRC32C over the 8 length bytes and compares it with the stored mask. A mismatch means the header is corrupted, so ValueError is raised including the hex of the header bytes.

Source

Thrown at sdks/python/apache_beam/io/tfrecordio.py:160

    Returns:
      None if EOF is reached; the paylod of the record otherwise.
    Raises:
      ValueError: If file appears to not be a valid TFRecords file.
    """
    buf_length_expected = 12
    buf = file_handle.read(buf_length_expected)
    if not buf:
      return None  # EOF Reached.

    # Validate all length related payloads.
    if len(buf) != buf_length_expected:
      raise ValueError(
          'Not a valid TFRecord. Fewer than %d bytes: %s' %
          (buf_length_expected, codecs.encode(buf, 'hex')))
    length, length_mask_expected = struct.unpack('<QI', buf)
    length_mask_actual = cls._masked_crc32c(buf[:8])
    if length_mask_actual != length_mask_expected:
      raise ValueError(
          'Not a valid TFRecord. Mismatch of length mask: %s' %
          codecs.encode(buf, 'hex'))

    # Validate all data related payloads.
    buf_length_expected = length + 4
    buf = file_handle.read(buf_length_expected)
    if len(buf) != buf_length_expected:
      raise ValueError(
          'Not a valid TFRecord. Fewer than %d bytes: %s' %
          (buf_length_expected, codecs.encode(buf, 'hex')))
    data, data_mask_expected = struct.unpack('<%dsI' % length, buf)
    data_mask_actual = cls._masked_crc32c(data)
    if data_mask_actual != data_mask_expected:
      raise ValueError(
          'Not a valid TFRecord. Mismatch of data mask: %s' %
          codecs.encode(buf, 'hex'))

    # All validation checks passed.

View on GitHub (pinned to 12126d8942)

Solutions

  1. Re-generate the TFRecord with a standard TensorFlow or Beam writer
  2. Re-transfer/re-upload the file and retry
  3. Compute the masked CRC yourself on a small file to confirm corruption before reporting upstream

Example fix

// before
# corrupted header file read as-is
// after
# re-write records:
with tf.io.TFRecordWriter(path) as w: w.write(record_bytes)
Defensive patterns

Strategy: validation

Validate before calling

import codecs, struct
with open(path, 'rb') as f:
    head = f.read(12)
length, mask = struct.unpack('<QI', head)
# recompute masked crc32c and compare before ingestion

Try / catch

try:
    pc | beam.io.ReadFromTFRecord(pattern)
except ValueError as e:
    if 'Mismatch of length mask' in str(e):
        raise RuntimeError('Corrupted TFRecord header in %s' % pattern) from e
    raise

Prevention

When it happens

Trigger: Reading a TFRecord whose 12-byte header was corrupted in transit/storage, or whose file was produced with an incompatible checksum variant.

Common situations: Bit rot or partial uploads; files manipulated (spliced/edited) with a different TFRecord implementation.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a5f6b6387b1c300e. Report an issue: GitHub.