apache/beam · error · ValueError

Not a valid TFRecord. Mismatch of data mask: %s

Error message

Not a valid TFRecord. Mismatch of data mask: %s

What it means

The reader computes masked CRC32C over the record payload and compares it to the 4-byte mask stored after the data. A mismatch indicates the payload bytes are corrupted, so ValueError is raised with hex of the header buffer (message text uses the header hex in this source, though conceptually it flags the data mask).

Source

Thrown at sdks/python/apache_beam/io/tfrecordio.py:174

          (buf_length_expected, codecs.encode(buf, 'hex')))
    length, length_mask_expected = struct.unpack('<QI', buf)
    length_mask_actual = cls._masked_crc32c(buf[:8])
    if length_mask_actual != length_mask_expected:
      raise ValueError(
          'Not a valid TFRecord. Mismatch of length mask: %s' %
          codecs.encode(buf, 'hex'))

    # Validate all data related payloads.
    buf_length_expected = length + 4
    buf = file_handle.read(buf_length_expected)
    if len(buf) != buf_length_expected:
      raise ValueError(
          'Not a valid TFRecord. Fewer than %d bytes: %s' %
          (buf_length_expected, codecs.encode(buf, 'hex')))
    data, data_mask_expected = struct.unpack('<%dsI' % length, buf)
    data_mask_actual = cls._masked_crc32c(data)
    if data_mask_actual != data_mask_expected:
      raise ValueError(
          'Not a valid TFRecord. Mismatch of data mask: %s' %
          codecs.encode(buf, 'hex'))

    # All validation checks passed.
    return data


class _TFRecordSource(FileBasedSource):
  """A File source for reading files of TFRecords.

  For detailed TFRecords format description see:
    https://www.tensorflow.org/versions/r1.11/api_guides/python/python_io#TFRecords_Format_Details
  """
  def __init__(self, file_pattern, coder, compression_type, validate):
    """Initialize a TFRecordSource.  See ReadFromTFRecord for details."""
    super().__init__(
        file_pattern=file_pattern,
        compression_type=compression_type,

View on GitHub (pinned to 12126d8942)

Solutions

1. Regenerate the TFRecord from source data
2. Restore the file from a backup
3. Check the storage medium / transfer path for corruption

Example fix

// before
# reading corrupted payload
// after
# rewrite from source:
with tf.io.TFRecordWriter(out_path) as w: [w.write(r) for r in records]

# before ingestion, verify data CRC on a sample record with masked_crc32c
from apache_beam.io import tfrecordio  # or compute crc32c directly
crc = tfrecordio._TFRecordUtil._masked_crc32c(data)
assert isinstance(crc, int)

try:
    pc | beam.io.ReadFromTFRecord(pattern)
except ValueError as e:
    if 'Mismatch of data mask' in str(e):
        raise RuntimeError('Corrupted record payload in %s' % pattern) from e
    raise

When it happens

Trigger: Reading a TFRecord whose payload bytes were altered after writing (bit flips, bad compression pipeline, wrong byte patching).

Common situations: Storage corruption, manual byte-level edits, or files concatenated incorrectly from different writers.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Not a valid TFRecord. Fewer than %d bytes: %s
• Not a valid TFRecord. Mismatch of length mask: %s
• Mismatch of length mask when reading a record. Expected %d b
• length overflow %d
• Mismatch of data mask when reading a record. Expected %d but

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/23dcfac0295c5eef. Report an issue: GitHub.