{"record":{"id":"0dfb9f14da32f579","repo":"apache/hadoop","slug":"invalid-size-size","errorCode":null,"errorMessage":"Invalid size: {size}","messagePattern":"Invalid size: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ObjectWritable.java","lineNumber":369,"sourceCode":"  private static Message tryInstantiateProtobuf(\n      Class<?> protoClass,\n      DataInput dataIn) throws IOException {\n\n    try {\n      if (dataIn instanceof InputStream) {\n        // We can use the built-in parseDelimitedFrom and not have to re-copy\n        // the data\n        Method parseMethod = getStaticProtobufMethod(protoClass,\n            \"parseDelimitedFrom\", InputStream.class);\n        return (Message)parseMethod.invoke(null, (InputStream)dataIn);\n      } else {\n        // Have to read it into a buffer first, since protobuf doesn't deal\n        // with the DataInput interface directly.\n        \n        // Read the size delimiter that writeDelimitedTo writes\n        int size = ProtoUtil.readRawVarint32(dataIn);\n        if (size < 0) {\n          throw new IOException(\"Invalid size: \" + size);\n        }\n      \n        byte[] data = new byte[size];\n        dataIn.readFully(data);\n        Method parseMethod = getStaticProtobufMethod(protoClass,\n            \"parseFrom\", byte[].class);\n        return (Message)parseMethod.invoke(null, data);\n      }\n    } catch (InvocationTargetException e) {\n      \n      if (e.getCause() instanceof IOException) {\n        throw (IOException)e.getCause();\n      } else {\n        throw new IOException(e.getCause());\n      }\n    } catch (IllegalAccessException iae) {\n      throw new AssertionError(\"Could not access parse method in \" +\n          protoClass);","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ObjectWritable.java#L351-L387","documentation":"Thrown in ObjectWritable's protobuf reading path (loadProtocol/ ObjectWritable.readObject for Message classes) when the size delimiter parsed via ProtoUtil.readRawVarint32(dataIn) is negative. writeDelimitedTo prefixes each message with an unsigned varint length; a negative decode means the varint consumed more than 32 bits worth of payload length — i.e. the stream is not where a delimited protobuf message starts. The bytes at the current position are not a length prefix, so the data is framed incorrectly or corrupted.","triggerScenarios":"Reading protobuf via ObjectWritable when the writer did NOT use writeDelimitedTo framing in the way the reader assumes (raw serialize vs delimited mismatch); stream position misaligned because a preceding field was mis-read; corrupted bytes (truncated or bit-flipped) inflating the varint; protobuf runtime version differences changing varint decoding of >32-bit values.","commonSituations":"Mixing protobuf serialization modes (serialize vs writeDelimitedTo) between Hadoop versions or with/without the direct-parse fast path; version skew between client and server protobuf versions; hand-implemented DataInput adapters feeding garbage to the protobuf reader.","solutions":["Ensure writer and reader run the same Hadoop/protobuf pair so both agree on delimited framing and the InputStream fast-path selection.","Verify the stream is positioned exactly where the delimited message begins — check the preceding field's read logic.","Check data integrity end-to-end (checksums, transfer layer) if framing code is known-consistent.","If you control both ends, prefer registering the protobuf class consistently and let ObjectWritable handle framing rather than interleaving raw writes."],"exampleFix":"// before: writer emits raw serialize(), reader expects delimited varint\nmsg.writeTo(dataOut);              // no length prefix\n// reader side: ProtoUtil.readRawVarint32 reads message bytes as a varint -> Invalid size\n\n// after: use delimited framing on both ends\nmsg.writeDelimitedTo(DataOutputOutputStream.constructOutputStream(dataOut));\n// reader: ObjectWritable/readObject now parses the varint length correctly","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  Object v = ObjectWritable.readObject(in, declaredClass, conf);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Invalid size\")) {\n    // framing mismatch: writer did not use delimited protobuf framing,\n    // or stream is misaligned/corrupt — do not retry the same stream\n    throw new IOException(\"Protobuf framing mismatch reading ObjectWritable\", e);\n  }\n  throw e;\n}","preventionTips":["Pin identical Hadoop and protobuf versions on both ends of any ObjectWritable channel.","Never mix msg.writeTo(dataOut) with the delimited path — always writeDelimitedTo for ObjectWritable-embedded messages.","Integration-test the exact writer/reader pair with large messages (>4KB) so varint framing is exercised.","On this error, discard the stream — resyncing mid-protobuf is unreliable by design."],"tags":["serialization","protobuf","framing","hadoop-common"],"backgroundTag":"message-framing-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}