apache/beam · error · RuntimeError

VarLong too long.

Error message

VarLong too long.

What it means

Raised by InnerStream.read_var_int64 when the varint occupies more bytes than a 64-bit integer allows: the shift reaches 64, or reaches 63 with more than 1 significant bit left. This indicates the input is not a valid 64-bit varint — the byte stream is corrupt or was encoded by an incompatible encoder. Beam aborts rather than silently overflowing.

Source

Thrown at sdks/python/apache_beam/coders/slow_stream.py:152

  def read_all(self, nested: bool) -> bytes:
    return self.read(self.read_var_int64() if nested else self.size())

  def read_byte(self) -> int:
    self.pos += 1
    return self.data[self.pos - 1]

  def read_var_int64(self):
    shift = 0
    result = 0
    while True:
      byte = self.read_byte()
      if byte < 0:
        raise RuntimeError('VarLong not terminated.')

      bits = byte & 0x7F
      if shift >= 64 or (shift >= 63 and bits > 1):
        raise RuntimeError('VarLong too long.')
      result |= bits << shift
      shift += 7
      if not byte & 0x80:
        break
    if result >= 1 << 63:
      result -= 1 << 64
    return result

  def read_var_int32(self):
    v = self.read_var_int64()
    return struct.unpack('<i', struct.pack('<I', v))[0]

  def read_bigendian_int64(self):
    return struct.unpack('>q', self.read(8))[0]

  def read_bigendian_uint64(self):
    return struct.unpack('>Q', self.read(8))[0]

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that decoding starts at a correct record boundary — a misaligned offset makes any bytes look like a giant varint.
2. Verify the data was encoded with Beam's varint encoder (max 10 bytes for int64).
3. Validate or checksum input payloads before decoding.
4. Catch RuntimeError and treat the stream as corrupt; skip or re-ingest the bad record.

Example fix

// before
pos = find_record(blob)  # may be wrong
value = stream_at(blob, pos).read_var_int64()
// after
pos = find_record(blob)
assert blob[pos:pos+10].count(0x80) < 10, 'implausible varint at %d' % pos
value = stream_at(blob, pos).read_var_int64()

head = data[pos:pos+10]
if head and head[-1] & 0x80:
    raise ValueError('varint exceeds 10 bytes at offset %d' % pos)

def is_plausible_varint(data: bytes, pos: int) -> bool:
    chunk = data[pos:pos+10]
    return bool(chunk) and not (chunk[-1] & 0x80)

try:
    value = stream.read_var_int64()
except RuntimeError as e:
    if 'VarLong too long' in str(e):
        resync_to_next_record()  # fix offset alignment
    else:
        raise

When it happens

Trigger: Decoding bytes whose varint continuation bits never terminate within 10 bytes, e.g. decoding non-Beam binary data as a varint, bit-rotted/corrupted payloads, or an offset mistake landing in the middle of unrelated binary data.

Common situations: Custom runners or DoFn tests decoding arbitrary binary blobs; parsing concatenated records where a length was misread so decoding drifts into wrong byte positions; fuzz or malformed-input handling.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• VarLong not terminated.
• varint overflow
• varint too long
• Encountered unexpected value for null indicator: '%s'
• Can not encode {} as a 64-bit integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3cfeb392eece9956. Report an issue: GitHub.