{"record":{"id":"508b453b071fbc7a","repo":"apache/beam","slug":"varlong-not-terminated","errorCode":null,"errorMessage":"VarLong not terminated.","messagePattern":"VarLong not terminated\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/coders/slow_stream.py","lineNumber":148,"sourceCode":"\n  def read(self, size: int) -> bytes:\n    self.pos += size\n    return self.data[self.pos - size:self.pos]\n\n  def read_all(self, nested: bool) -> bytes:\n    return self.read(self.read_var_int64() if nested else self.size())\n\n  def read_byte(self) -> int:\n    self.pos += 1\n    return self.data[self.pos - 1]\n\n  def read_var_int64(self):\n    shift = 0\n    result = 0\n    while True:\n      byte = self.read_byte()\n      if byte < 0:\n        raise RuntimeError('VarLong not terminated.')\n\n      bits = byte & 0x7F\n      if shift >= 64 or (shift >= 63 and bits > 1):\n        raise RuntimeError('VarLong too long.')\n      result |= bits << shift\n      shift += 7\n      if not byte & 0x80:\n        break\n    if result >= 1 << 63:\n      result -= 1 << 64\n    return result\n\n  def read_var_int32(self):\n    v = self.read_var_int64()\n    return struct.unpack('<i', struct.pack('<I', v))[0]\n\n  def read_bigendian_int64(self):\n    return struct.unpack('>q', self.read(8))[0]","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/coders/slow_stream.py#L130-L166","documentation":"Raised by InnerStream.read_var_int64 when a varint-encoded integer stream hits EOF before a terminating byte (one with the high bit clear) is read. This means the byte stream is truncated or corrupted: the VarLong encoding requires every byte except the last to have its continuation bit set. Beam throws it as a RuntimeError because continuing to decode would silently produce wrong values.","triggerScenarios":"Calling read_var_int64 (directly or via read_var_int32/read_all) on a stream that ends mid-varint — e.g. a truncated coder output, an incorrect offset/limit read of encoded data, or decoding data not produced by Beam's varint encoder.","commonSituations":"Reading partially-downloaded or truncated encoded coder blobs; hand-rolled wire-protocol parsing where a length prefix doesn't match the actual payload; mixups between compressed and decompressed streams in custom runners or test harnesses.","solutions":["Verify the source data is complete and was fully written by the encoder (check byte counts/length prefixes).","Confirm you are decoding data encoded with Beam's varint encoding (encode_var_int64), not some other integer format.","Read through a stream that delivers all bytes; check for early-close or short reads in your custom Stream implementation.","Wrap decode in try/except RuntimeError and treat the input as corrupt — re-encode or re-fetch the data."],"exampleFix":"// before\nstream = open('partial_output.beam', 'rb')\nvalue = slow_stream.BigDecimalStream(stream).read_var_int64()\n// after\nraw = open('output.beam', 'rb').read()\nassert len(raw) == expected_length, 'truncated input'\nvalue = slow_stream.BigDecimalStream(io.BytesIO(raw)).read_var_int64()","handlingStrategy":"try-catch","validationCode":"data = src.read(expected_len)\nif len(data) < expected_len:\n    raise ValueError('truncated varint stream')\nif data and not (data[-1] & 0x80):\n    pass  # last byte terminates the varint\nelse:\n    raise ValueError('varint not terminated')","typeGuard":"def is_complete_varint(data: bytes) -> bool:\n    return bool(data) and not (data[-1] & 0x80)","tryCatchPattern":"try:\n    value = stream.read_var_int64()\nexcept RuntimeError as e:\n    if 'VarLong not terminated' in str(e):\n        handle_corrupt_stream()  # re-fetch or skip record\n    else:\n        raise","preventionTips":["Always consume encoded data written as a complete unit; check length prefixes.","Ensure streams are not closed or truncated before decoding finishes.","Decode only data produced by Beam's varint encoder.","Checksum payloads transferred across processes/networks."],"tags":["python","decoding","corrupt-data","varint"],"backgroundTag":"unexpected-response-shape","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}