apache/beam · error · RuntimeError

VarInt not terminated.

Error message

VarInt not terminated.

What it means

read_var_int64 keeps consuming bytes while the continuation bit (0x80) is set; read_byte() returning a negative value signals end-of-stream. If EOF is hit before a terminating byte (0x00–0x7F) is seen, the coder raises RuntimeError('VarInt not terminated.') because the encoded integer is truncated.

Solutions

1. Check the source for truncation (file size, shard completeness) and re-read/rewrite the data
2. Ensure the writer flushed/closed the stream before the reader consumed it
3. Verify buffer framing: don't call read_var_int64 past the end of a length-delimited record
4. Catch the RuntimeError and treat the stream as corrupt/ended rather than retrying the read

Example fix

# before
value = stream.read_var_int64()  # raises on truncated tail
# after
try:
    value = stream.read_var_int64()
except RuntimeError as e:
    if 'VarInt not terminated' in str(e):
        handle_truncated_stream()  # stop reading, mark source incomplete
    else:
        raise

try:
    value = stream.read_var_int64()
except RuntimeError as e:
    if 'VarInt not terminated' in str(e):
        handle_truncated_input()  # stop, mark incomplete
    else:
        raise

When it happens

Trigger: Reading a truncated stream — a varint's bytes are cut off by EOF (truncated file/shard, closed gRPC stream, short read) while the last byte still has the continuation bit set.

Common situations: Partially written output files read before flush/close; network streams cut mid-record; wrong buffer sizing causing reads past the record boundary.

Related errors

• VarLong too long.
• Bad coder for input of
• Bad coder for output of
• 'Cannot provide coder for
• Coder registration requires a coder class object. Received…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/26a18488ccb79bb6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/coders/stream.pyx:236

    self.pos += 1
    if byte <= 0x7F:
      return byte

    cdef libc.stdint.int64_t bits
    cdef long shift = 0
    cdef libc.stdint.int64_t result = 0
    while True:
      bits = byte & 0x7F
      if (shift >= sizeof(libc.stdint.int64_t) * 8 or
          (shift >= (sizeof(libc.stdint.int64_t) * 8 - 1) and bits > 1)):
        raise RuntimeError('VarLong too long.')
      result |= bits << shift
      shift += 7
      if not (byte & 0x80):
        break
      byte = self.read_byte()
      if byte < 0:
        raise RuntimeError('VarInt not terminated.')

    return result

  cpdef libc.stdint.int32_t read_var_int32(self) except? -1:
    """Decode a variable-length encoded int32 from a stream."""
    cdef libc.stdint.int64_t v = self.read_var_int64()
    return <libc.stdint.int32_t>(v);

  cpdef libc.stdint.int64_t read_bigendian_int64(self) except? -1:
    return self.read_bigendian_uint64()

  cpdef libc.stdint.uint64_t read_bigendian_uint64(self) except? -1:
    self.pos += 8
    return (<unsigned char>self.allc[self.pos - 1]
      | <libc.stdint.uint64_t><unsigned char>self.allc[self.pos - 2] <<  8
      | <libc.stdint.uint64_t><unsigned char>self.allc[self.pos - 3] << 16
      | <libc.stdint.uint64_t><unsigned char>self.allc[self.pos - 4] << 24
      | <libc.stdint.uint64_t><unsigned char>self.allc[self.pos - 5] << 32

View on GitHub (pinned to 12126d8942)