{"record":{"id":"aeffb92bce5ed7ed","repo":"apache/beam","slug":"not-a-valid-tfrecord-fewer-than-d-bytes-s","errorCode":null,"errorMessage":"Not a valid TFRecord. Fewer than %d bytes: %s","messagePattern":"Not a valid TFRecord\\. Fewer than (.+?) bytes: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/tfrecordio.py","lineNumber":154,"sourceCode":"  @classmethod\n  def read_record(cls, file_handle):\n    \"\"\"Read a record from a TFRecords file.\n\n    Args:\n      file_handle: The file to read from.\n    Returns:\n      None if EOF is reached; the paylod of the record otherwise.\n    Raises:\n      ValueError: If file appears to not be a valid TFRecords file.\n    \"\"\"\n    buf_length_expected = 12\n    buf = file_handle.read(buf_length_expected)\n    if not buf:\n      return None  # EOF Reached.\n\n    # Validate all length related payloads.\n    if len(buf) != buf_length_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Fewer than %d bytes: %s' %\n          (buf_length_expected, codecs.encode(buf, 'hex')))\n    length, length_mask_expected = struct.unpack('<QI', buf)\n    length_mask_actual = cls._masked_crc32c(buf[:8])\n    if length_mask_actual != length_mask_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Mismatch of length mask: %s' %\n          codecs.encode(buf, 'hex'))\n\n    # Validate all data related payloads.\n    buf_length_expected = length + 4\n    buf = file_handle.read(buf_length_expected)\n    if len(buf) != buf_length_expected:\n      raise ValueError(\n          'Not a valid TFRecord. Fewer than %d bytes: %s' %\n          (buf_length_expected, codecs.encode(buf, 'hex')))\n    data, data_mask_expected = struct.unpack('<%dsI' % length, buf)\n    data_mask_actual = cls._masked_crc32c(data)","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/tfrecordio.py#L136-L172","documentation":"While reading a TFRecord, the reader first expects a 12-byte header (8-byte length + 4-byte masked CRC). If fewer than that many bytes are read yet the buffer is non-empty, the record framing is corrupt and ValueError is raised with the expected size and hex of the truncated bytes.","triggerScenarios":"Reading a file that is not a TFRecord, or a TFRecord truncated mid-header, via ReadFromTFRecord.","commonSituations":"Pointing ReadFromTFRecord at a plain text/Avro/JSON file; partial or corrupted uploads to GCS/S3; files cut off by failed writes.","solutions":["Verify the file is a real TFRecord produced by a TFRecord writer","Re-export or re-upload the corrupted/truncated file","Check that the correct file pattern is being read (not a mixed directory)"],"exampleFix":"// before\nReadFromTFRecord('gs://bucket/logs/*.txt')  # not TFRecord\n// after\n# write data with tf.io.TFRecordWriter first, then:\nReadFromTFRecord('gs://bucket/records/*.tfrecord')","handlingStrategy":"validation","validationCode":"def looks_like_tfrecord(path) -> bool:\n    import struct\n    with open(path, 'rb') as f:\n        head = f.read(12)\n    return len(head) == 12 and len(struct.unpack('<Q', head[:8])[0]) >= 0 if head else False\n# better: check file magic/size before handing to ReadFromTFRecord","typeGuard":null,"tryCatchPattern":"try:\n    pc | beam.io.ReadFromTFRecord(pattern)\nexcept ValueError as e:\n    if 'Not a valid TFRecord' in str(e):\n        raise RuntimeError('File %s is truncated or not a TFRecord' % pattern) from e\n    raise","preventionTips":["Only read files produced by tf.io.TFRecordWriter or Beam's WriteToTFRecord","Ensure uploads complete before reading (use GCS finalize events)","Validate file sizes against the writer's expectations"],"tags":["python","beam","tfrecord","corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}