apache/beam · error · IOException

varint overflow

Error message

varint overflow 

What it means

decodeInt reads a base-128 varint from a stream and requires the value to fit in a signed 32-bit int (0 <= r < 2^32 as an unsigned reading). If the decoded long is negative or >= 2^32, the value cannot be represented as an int and an IOException is thrown.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java:107

      stream.write((byte) v);
      return;
    }
    stream.write((byte) (v | 0x80));
    v >>>= 7;
    if ((v & ~0x7F) == 0) {
      stream.write((byte) v);
      return;
    }
    stream.write((byte) (v | 0x80));
    v >>>= 7;
    stream.write((byte) v);
  }

  /** Decodes an integer value from the given stream. */
  public static int decodeInt(InputStream stream) throws IOException {
    long r = decodeLong(stream);
    if (r < 0 || r >= 1L << 32) {
      throw new IOException("varint overflow " + r);
    }
    return (int) r;
  }

  /** Decodes a long value from the given stream. */
  public static long decodeLong(InputStream stream) throws IOException {
    long result = 0;
    int shift = 0;
    int b;
    do {
      // Get 7 bits from next byte
      b = stream.read();
      if (b < 0) {
        if (shift == 0) {
          throw new EOFException();
        } else {
          throw new IOException("varint not terminated");
        }

View on GitHub (pinned to 12126d8942)

Solutions

1. Use VarInt.decodeLong instead of decodeInt when the value may exceed int range
2. Fix the writer side to use encodeInt, or encode values that fit in 32 bits
3. Verify stream alignment: re-check the serialization framing so decoding starts at the correct byte
4. Validate decoded values at write time before persisting

Example fix

// before
int n = VarInt.decodeInt(stream);
// after
long r = VarInt.decodeLong(stream);
if (r < 0 || r >= 1L << 32) throw new IOException("value out of int range: " + r);
int n = (int) r;

// on the writer side, before encoding:
if (value < 0 || value >= 1L << 32) throw new IllegalArgumentException("value does not fit int varint: " + value);

try { int n = VarInt.decodeInt(stream); } catch (IOException e) { if (e.getMessage().startsWith("varint overflow")) { /* re-read with decodeLong */ } }

When it happens

Trigger: Reading a varint that was encoded from a long value outside the int range (negative longs, or values >= 2^32), or reading a corrupted/misaligned stream where varint bytes shift decoding into a huge value.

Common situations: Decoding data written with VarInt.encodeLong (or another encoder allowing 64-bit values) with decodeInt; deserializing a stream whose field boundaries were lost so bytes of two varints merge into one; schema/version mismatches between writer and reader.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• varint too long
• varint not terminated
• NullableCoder expects either a byte valued 0 (null) or 1 (pr
• unable to deserialize record
• Invalid encoded string length: {}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c9a3f9ea9526cf55. Report an issue: GitHub.