{"record":{"id":"a5f6b6387b1c300e","repo":"apache/beam","slug":"not-a-valid-tfrecord-mismatch-of-length-mask-s","errorCode":null,"errorMessage":"Not a valid TFRecord. Mismatch of length mask: %s","messagePattern":"Not a valid TFRecord\\. Mismatch of length mask: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/tfrecordio.py","lineNumber":160,"sourceCode":"    Returns:\n      None if EOF is reached; the paylod of the record otherwise.\n    Raises:\n      ValueError: If file appears to not be a valid TFRecords file.\n    \"\"\"\n    buf_length_expected = 12\n    buf = file_handle.read(buf_length_expected)\n    if not buf:\n      return None  # EOF Reached.\n\n    # Validate all length related payloads.\n    if len(buf) != buf_length_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Fewer than %d bytes: %s' %\n          (buf_length_expected, codecs.encode(buf, 'hex')))\n    length, length_mask_expected = struct.unpack('<QI', buf)\n    length_mask_actual = cls._masked_crc32c(buf[:8])\n    if length_mask_actual != length_mask_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Mismatch of length mask: %s' %\n          codecs.encode(buf, 'hex'))\n\n    # Validate all data related payloads.\n    buf_length_expected = length + 4\n    buf = file_handle.read(buf_length_expected)\n    if len(buf) != buf_length_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Fewer than %d bytes: %s' %\n          (buf_length_expected, codecs.encode(buf, 'hex')))\n    data, data_mask_expected = struct.unpack('<%dsI' % length, buf)\n    data_mask_actual = cls._masked_crc32c(data)\n    if data_mask_actual != data_mask_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Mismatch of data mask: %s' %\n          codecs.encode(buf, 'hex'))\n\n    # All validation checks passed.","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/tfrecordio.py#L142-L178","documentation":"After unpacking the length header, the reader recomputes the masked CRC32C over the 8 length bytes and compares it with the stored mask. A mismatch means the header is corrupted, so ValueError is raised including the hex of the header bytes.","triggerScenarios":"Reading a TFRecord whose 12-byte header was corrupted in transit/storage, or whose file was produced with an incompatible checksum variant.","commonSituations":"Bit rot or partial uploads; files manipulated (spliced/edited) with a different TFRecord implementation.","solutions":["Re-generate the TFRecord with a standard TensorFlow or Beam writer","Re-transfer/re-upload the file and retry","Compute the masked CRC yourself on a small file to confirm corruption before reporting upstream"],"exampleFix":"// before\n# corrupted header file read as-is\n// after\n# re-write records:\nwith tf.io.TFRecordWriter(path) as w: w.write(record_bytes)","handlingStrategy":"validation","validationCode":"import codecs, struct\nwith open(path, 'rb') as f:\n    head = f.read(12)\nlength, mask = struct.unpack('<QI', head)\n# recompute masked crc32c and compare before ingestion","typeGuard":null,"tryCatchPattern":"try:\n    pc | beam.io.ReadFromTFRecord(pattern)\nexcept ValueError as e:\n    if 'Mismatch of length mask' in str(e):\n        raise RuntimeError('Corrupted TFRecord header in %s' % pattern) from e\n    raise","preventionTips":["Avoid manual byte-level edits to TFRecord files","Verify checksums after network transfer","Restore from backup instead of patching corrupted files"],"tags":["python","beam","tfrecord","corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}