apache/hadoop · error · IOException

Invalid size: {size}

Error message

Invalid size: {size}

What it means

Thrown in ObjectWritable's protobuf reading path (loadProtocol/ ObjectWritable.readObject for Message classes) when the size delimiter parsed via ProtoUtil.readRawVarint32(dataIn) is negative. writeDelimitedTo prefixes each message with an unsigned varint length; a negative decode means the varint consumed more than 32 bits worth of payload length — i.e. the stream is not where a delimited protobuf message starts. The bytes at the current position are not a length prefix, so the data is framed incorrectly or corrupted.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ObjectWritable.java:369

  private static Message tryInstantiateProtobuf(
      Class<?> protoClass,
      DataInput dataIn) throws IOException {

    try {
      if (dataIn instanceof InputStream) {
        // We can use the built-in parseDelimitedFrom and not have to re-copy
        // the data
        Method parseMethod = getStaticProtobufMethod(protoClass,
            "parseDelimitedFrom", InputStream.class);
        return (Message)parseMethod.invoke(null, (InputStream)dataIn);
      } else {
        // Have to read it into a buffer first, since protobuf doesn't deal
        // with the DataInput interface directly.
        
        // Read the size delimiter that writeDelimitedTo writes
        int size = ProtoUtil.readRawVarint32(dataIn);
        if (size < 0) {
          throw new IOException("Invalid size: " + size);
        }
      
        byte[] data = new byte[size];
        dataIn.readFully(data);
        Method parseMethod = getStaticProtobufMethod(protoClass,
            "parseFrom", byte[].class);
        return (Message)parseMethod.invoke(null, data);
      }
    } catch (InvocationTargetException e) {
      
      if (e.getCause() instanceof IOException) {
        throw (IOException)e.getCause();
      } else {
        throw new IOException(e.getCause());
      }
    } catch (IllegalAccessException iae) {
      throw new AssertionError("Could not access parse method in " +
          protoClass);

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure writer and reader run the same Hadoop/protobuf pair so both agree on delimited framing and the InputStream fast-path selection.
2. Verify the stream is positioned exactly where the delimited message begins — check the preceding field's read logic.
3. Check data integrity end-to-end (checksums, transfer layer) if framing code is known-consistent.
4. If you control both ends, prefer registering the protobuf class consistently and let ObjectWritable handle framing rather than interleaving raw writes.

Example fix

// before: writer emits raw serialize(), reader expects delimited varint
msg.writeTo(dataOut);              // no length prefix
// reader side: ProtoUtil.readRawVarint32 reads message bytes as a varint -> Invalid size

// after: use delimited framing on both ends
msg.writeDelimitedTo(DataOutputOutputStream.constructOutputStream(dataOut));
// reader: ObjectWritable/readObject now parses the varint length correctly

try {
  Object v = ObjectWritable.readObject(in, declaredClass, conf);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Invalid size")) {
    // framing mismatch: writer did not use delimited protobuf framing,
    // or stream is misaligned/corrupt — do not retry the same stream
    throw new IOException("Protobuf framing mismatch reading ObjectWritable", e);
  }
  throw e;
}

When it happens

Trigger: Reading protobuf via ObjectWritable when the writer did NOT use writeDelimitedTo framing in the way the reader assumes (raw serialize vs delimited mismatch); stream position misaligned because a preceding field was mis-read; corrupted bytes (truncated or bit-flipped) inflating the varint; protobuf runtime version differences changing varint decoding of >32-bit values.

Common situations: Mixing protobuf serialization modes (serialize vs writeDelimitedTo) between Hadoop versions or with/without the direct-parse fast path; version skew between client and server protobuf versions; hand-implemented DataInput adapters feeding garbage to the protobuf reader.

Related errors

• Can't read FileStatusProto with negative size of ${size}
• Not a primitive: {declaredClass}
• Can't write: {instance} as {declaredClass}
• readObject can't find class {className}
• Cannot wrap ${class}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0dfb9f14da32f579. Report an issue: GitHub.