apache/beam · error · IOException

varint too long

Error message

varint too long

What it means

decodeLong stops after it has consumed enough bits for a 64-bit value: once shift >= 64 (or shift == 63 with more than the final bit set), the varint cannot fit in a long. This means the input is either not a valid varint or encodes a value exceeding 64 bits, so an IOException 'varint too long' is thrown.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java:130

  /** Decodes a long value from the given stream. */
  public static long decodeLong(InputStream stream) throws IOException {
    long result = 0;
    int shift = 0;
    int b;
    do {
      // Get 7 bits from next byte
      b = stream.read();
      if (b < 0) {
        if (shift == 0) {
          throw new EOFException();
        } else {
          throw new IOException("varint not terminated");
        }
      }
      long bits = b & 0x7F;
      if (shift >= 64 || (shift == 63 && bits > 1)) {
        // Out of range
        throw new IOException("varint too long");
      }
      result |= bits << shift;
      shift += 7;
    } while ((b & 0x80) != 0);
    return result;
  }

  /** Returns the length of the encoding of the given value (in bytes). */
  public static int getLength(int v) {
    return CodedOutputStream.computeUInt32SizeNoTag(v);
  }

  /** Returns the length of the encoding of the given value (in bytes). */
  public static int getLength(long v) {
    return CodedOutputStream.computeUInt64SizeNoTag(v);
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Confirm writer and reader use compatible varint encodings (same width, same field order)
2. Fix stream alignment: the decode position must be exactly at a varint start; re-sync framing
3. If values can exceed 64 bits, use BigInteger decoding instead of VarInt.decodeLong
4. Validate/clean the input data source for corruption

Example fix

// before
long v = VarInt.decodeLong(stream); // throws if > 64 bits
// after
long v = readVarIntAsBigInteger(stream).longValueExact(); // handles wider values explicitly

// cap continuation bytes before decoding: count bytes with high bit set; if > 10, abort

try { long v = VarInt.decodeLong(stream); } catch (IOException e) { if (e.getMessage().equals("varint too long")) { /* stream misaligned or invalid encoder */ } }

When it happens

Trigger: Decoding a varint with 10+ continuation bytes (value >= 2^64); reading misaligned data where non-varint bytes are interpreted as varint continuation bytes (high bits set), causing the loop to run past 64 bits; corrupt input.

Common situations: Reading data written by a different/non-canonical varint encoder (e.g. protobuf of larger width or sign-extended oversized encodings); stream desync after an earlier bad decode; feeding binary data to the wrong parser.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• varint overflow
• varint not terminated
• NullableCoder expects either a byte valued 0 (null) or 1 (pr
• unable to deserialize record
• Invalid encoded string length: {}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3c1a3d72d504c4b8. Report an issue: GitHub.