{"record":{"id":"3cfeb392eece9956","repo":"apache/beam","slug":"varlong-too-long","errorCode":null,"errorMessage":"VarLong too long.","messagePattern":"VarLong too long\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/coders/slow_stream.py","lineNumber":152,"sourceCode":"\n  def read_all(self, nested: bool) -> bytes:\n    return self.read(self.read_var_int64() if nested else self.size())\n\n  def read_byte(self) -> int:\n    self.pos += 1\n    return self.data[self.pos - 1]\n\n  def read_var_int64(self):\n    shift = 0\n    result = 0\n    while True:\n      byte = self.read_byte()\n      if byte < 0:\n        raise RuntimeError('VarLong not terminated.')\n\n      bits = byte & 0x7F\n      if shift >= 64 or (shift >= 63 and bits > 1):\n        raise RuntimeError('VarLong too long.')\n      result |= bits << shift\n      shift += 7\n      if not byte & 0x80:\n        break\n    if result >= 1 << 63:\n      result -= 1 << 64\n    return result\n\n  def read_var_int32(self):\n    v = self.read_var_int64()\n    return struct.unpack('<i', struct.pack('<I', v))[0]\n\n  def read_bigendian_int64(self):\n    return struct.unpack('>q', self.read(8))[0]\n\n  def read_bigendian_uint64(self):\n    return struct.unpack('>Q', self.read(8))[0]\n","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/coders/slow_stream.py#L134-L170","documentation":"Raised by InnerStream.read_var_int64 when the varint occupies more bytes than a 64-bit integer allows: the shift reaches 64, or reaches 63 with more than 1 significant bit left. This indicates the input is not a valid 64-bit varint — the byte stream is corrupt or was encoded by an incompatible encoder. Beam aborts rather than silently overflowing.","triggerScenarios":"Decoding bytes whose varint continuation bits never terminate within 10 bytes, e.g. decoding non-Beam binary data as a varint, bit-rotted/corrupted payloads, or an offset mistake landing in the middle of unrelated binary data.","commonSituations":"Custom runners or DoFn tests decoding arbitrary binary blobs; parsing concatenated records where a length was misread so decoding drifts into wrong byte positions; fuzz or malformed-input handling.","solutions":["Check that decoding starts at a correct record boundary — a misaligned offset makes any bytes look like a giant varint.","Verify the data was encoded with Beam's varint encoder (max 10 bytes for int64).","Validate or checksum input payloads before decoding.","Catch RuntimeError and treat the stream as corrupt; skip or re-ingest the bad record."],"exampleFix":"// before\npos = find_record(blob)  # may be wrong\nvalue = stream_at(blob, pos).read_var_int64()\n// after\npos = find_record(blob)\nassert blob[pos:pos+10].count(0x80) < 10, 'implausible varint at %d' % pos\nvalue = stream_at(blob, pos).read_var_int64()","handlingStrategy":"try-catch","validationCode":"head = data[pos:pos+10]\nif head and head[-1] & 0x80:\n    raise ValueError('varint exceeds 10 bytes at offset %d' % pos)","typeGuard":"def is_plausible_varint(data: bytes, pos: int) -> bool:\n    chunk = data[pos:pos+10]\n    return bool(chunk) and not (chunk[-1] & 0x80)","tryCatchPattern":"try:\n    value = stream.read_var_int64()\nexcept RuntimeError as e:\n    if 'VarLong too long' in str(e):\n        resync_to_next_record()  # fix offset alignment\n    else:\n        raise","preventionTips":["Verify decoding starts at valid record boundaries.","Only decode varints encoded with Beam's encoder (<=10 bytes for int64).","Fuzz-test custom decoders against malformed input.","Log the byte offset to aid debugging misaligned parses."],"tags":["python","decoding","varint","overflow"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}