apache/beam · error · IOException

varint not terminated

Error message

varint not terminated

What it means

decodeLong reads a varint until it encounters a byte with the high bit clear. If the stream ends (read() returns -1) while mid-varint (some bytes already consumed, shift != 0), the encoding is truncated and an IOException 'varint not terminated' is thrown. If no bytes were read at all, an EOFException is thrown instead.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java:124

    if (r < 0 || r >= 1L << 32) {
      throw new IOException("varint overflow " + r);
    }
    return (int) r;
  }

  /** Decodes a long value from the given stream. */
  public static long decodeLong(InputStream stream) throws IOException {
    long result = 0;
    int shift = 0;
    int b;
    do {
      // Get 7 bits from next byte
      b = stream.read();
      if (b < 0) {
        if (shift == 0) {
          throw new EOFException();
        } else {
          throw new IOException("varint not terminated");
        }
      }
      long bits = b & 0x7F;
      if (shift >= 64 || (shift == 63 && bits > 1)) {
        // Out of range
        throw new IOException("varint too long");
      }
      result |= bits << shift;
      shift += 7;
    } while ((b & 0x80) != 0);
    return result;
  }

  /** Returns the length of the encoding of the given value (in bytes). */
  public static int getLength(int v) {
    return CodedOutputStream.computeUInt32SizeNoTag(v);
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that the source data is complete and uncorrupted; re-transfer or re-generate the input
2. Fix framing logic so decodeLong is only called when a full varint is available (e.g. read into a byte[] and bounds-check first)
3. Catch EOFException vs IOException to distinguish clean end-of-stream from truncation
4. Ensure writer flushes/closes completely before the reader consumes

Example fix

// before
long v = VarInt.decodeLong(stream); // may throw on truncation
// after
byte[] buf = readFully(stream, maxVarintBytes); // returns null on clean EOF
long v = (buf == null) ? -1 : VarInt.decodeLong(new ByteArrayInputStream(buf));

// check remaining data before decoding
if (available(stream) <= 0) return endOfStream; // custom bounds check on buffered input

try { long v = VarInt.decodeLong(stream); } catch (EOFException e) { /* clean end of stream */ } catch (IOException e) { /* truncated varint: input corrupt or framing bug */ }

When it happens

Trigger: Truncated input stream: the encoded data ends in the middle of a multi-byte varint; reading past the end of a record because length prefixes were misparsed; reading from a socket/file closed early.

Common situations: Corrupt or partially written files; network streams cut off mid-message; decoder/writer mismatch where a reader expects a varint at a position where the writer wrote none (schema drift); reading with the wrong offset after a failed earlier decode.

Related errors

• varint overflow
• varint too long
• NullableCoder expects either a byte valued 0 (null) or 1 (pr
• unable to deserialize record
• Invalid encoded string length: {}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b6672c2fd0b1dcdc. Report an issue: GitHub.