apache/beam · error · ValueError

Not a valid TFRecord. Fewer than %d bytes: %s

Error message

Not a valid TFRecord. Fewer than %d bytes: %s

What it means

While reading a TFRecord, the reader first expects a 12-byte header (8-byte length + 4-byte masked CRC). If fewer than that many bytes are read yet the buffer is non-empty, the record framing is corrupt and ValueError is raised with the expected size and hex of the truncated bytes.

Source

Thrown at sdks/python/apache_beam/io/tfrecordio.py:154

  @classmethod
  def read_record(cls, file_handle):
    """Read a record from a TFRecords file.

    Args:
      file_handle: The file to read from.
    Returns:
      None if EOF is reached; the paylod of the record otherwise.
    Raises:
      ValueError: If file appears to not be a valid TFRecords file.
    """
    buf_length_expected = 12
    buf = file_handle.read(buf_length_expected)
    if not buf:
      return None  # EOF Reached.

    # Validate all length related payloads.
    if len(buf) != buf_length_expected:
      raise ValueError(
          'Not a valid TFRecord. Fewer than %d bytes: %s' %
          (buf_length_expected, codecs.encode(buf, 'hex')))
    length, length_mask_expected = struct.unpack('<QI', buf)
    length_mask_actual = cls._masked_crc32c(buf[:8])
    if length_mask_actual != length_mask_expected:
      raise ValueError(
          'Not a valid TFRecord. Mismatch of length mask: %s' %
          codecs.encode(buf, 'hex'))

    # Validate all data related payloads.
    buf_length_expected = length + 4
    buf = file_handle.read(buf_length_expected)
    if len(buf) != buf_length_expected:
      raise ValueError(
          'Not a valid TFRecord. Fewer than %d bytes: %s' %
          (buf_length_expected, codecs.encode(buf, 'hex')))
    data, data_mask_expected = struct.unpack('<%dsI' % length, buf)
    data_mask_actual = cls._masked_crc32c(data)

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the file is a real TFRecord produced by a TFRecord writer
2. Re-export or re-upload the corrupted/truncated file
3. Check that the correct file pattern is being read (not a mixed directory)

Example fix

// before
ReadFromTFRecord('gs://bucket/logs/*.txt')  # not TFRecord
// after
# write data with tf.io.TFRecordWriter first, then:
ReadFromTFRecord('gs://bucket/records/*.tfrecord')

def looks_like_tfrecord(path) -> bool:
    import struct
    with open(path, 'rb') as f:
        head = f.read(12)
    return len(head) == 12 and len(struct.unpack('<Q', head[:8])[0]) >= 0 if head else False
# better: check file magic/size before handing to ReadFromTFRecord

try:
    pc | beam.io.ReadFromTFRecord(pattern)
except ValueError as e:
    if 'Not a valid TFRecord' in str(e):
        raise RuntimeError('File %s is truncated or not a TFRecord' % pattern) from e
    raise

When it happens

Trigger: Reading a file that is not a TFRecord, or a TFRecord truncated mid-header, via ReadFromTFRecord.

Common situations: Pointing ReadFromTFRecord at a plain text/Avro/JSON file; partial or corrupted uploads to GCS/S3; files cut off by failed writes.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Not a valid TFRecord. Mismatch of length mask: %s
• Not a valid TFRecord. Mismatch of data mask: %s
• Mismatch of length mask when reading a record. Expected %d b
• length overflow %d
• Mismatch of data mask when reading a record. Expected %d but

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aeffb92bce5ed7ed. Report an issue: GitHub.