apache/beam · error · RuntimeError

VarLong not terminated.

Error message

VarLong not terminated.

What it means

Raised by InnerStream.read_var_int64 when a varint-encoded integer stream hits EOF before a terminating byte (one with the high bit clear) is read. This means the byte stream is truncated or corrupted: the VarLong encoding requires every byte except the last to have its continuation bit set. Beam throws it as a RuntimeError because continuing to decode would silently produce wrong values.

Source

Thrown at sdks/python/apache_beam/coders/slow_stream.py:148

  def read(self, size: int) -> bytes:
    self.pos += size
    return self.data[self.pos - size:self.pos]

  def read_all(self, nested: bool) -> bytes:
    return self.read(self.read_var_int64() if nested else self.size())

  def read_byte(self) -> int:
    self.pos += 1
    return self.data[self.pos - 1]

  def read_var_int64(self):
    shift = 0
    result = 0
    while True:
      byte = self.read_byte()
      if byte < 0:
        raise RuntimeError('VarLong not terminated.')

      bits = byte & 0x7F
      if shift >= 64 or (shift >= 63 and bits > 1):
        raise RuntimeError('VarLong too long.')
      result |= bits << shift
      shift += 7
      if not byte & 0x80:
        break
    if result >= 1 << 63:
      result -= 1 << 64
    return result

  def read_var_int32(self):
    v = self.read_var_int64()
    return struct.unpack('<i', struct.pack('<I', v))[0]

  def read_bigendian_int64(self):
    return struct.unpack('>q', self.read(8))[0]

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the source data is complete and was fully written by the encoder (check byte counts/length prefixes).
2. Confirm you are decoding data encoded with Beam's varint encoding (encode_var_int64), not some other integer format.
3. Read through a stream that delivers all bytes; check for early-close or short reads in your custom Stream implementation.
4. Wrap decode in try/except RuntimeError and treat the input as corrupt — re-encode or re-fetch the data.

Example fix

// before
stream = open('partial_output.beam', 'rb')
value = slow_stream.BigDecimalStream(stream).read_var_int64()
// after
raw = open('output.beam', 'rb').read()
assert len(raw) == expected_length, 'truncated input'
value = slow_stream.BigDecimalStream(io.BytesIO(raw)).read_var_int64()

data = src.read(expected_len)
if len(data) < expected_len:
    raise ValueError('truncated varint stream')
if data and not (data[-1] & 0x80):
    pass  # last byte terminates the varint
else:
    raise ValueError('varint not terminated')

def is_complete_varint(data: bytes) -> bool:
    return bool(data) and not (data[-1] & 0x80)

try:
    value = stream.read_var_int64()
except RuntimeError as e:
    if 'VarLong not terminated' in str(e):
        handle_corrupt_stream()  # re-fetch or skip record
    else:
        raise

When it happens

Trigger: Calling read_var_int64 (directly or via read_var_int32/read_all) on a stream that ends mid-varint — e.g. a truncated coder output, an incorrect offset/limit read of encoded data, or decoding data not produced by Beam's varint encoder.

Common situations: Reading partially-downloaded or truncated encoded coder blobs; hand-rolled wire-protocol parsing where a length prefix doesn't match the actual payload; mixups between compressed and decompressed streams in custom runners or test harnesses.

Related errors

• VarLong too long.
• invalid varintz encoding for: %v
• invalid varuintz encoding for: %v
• invalid length " + length
• Invalid encoded byte array

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/508b453b071fbc7a. Report an issue: GitHub.