{"record":{"id":"5f5d346e17068191","repo":"apache/hadoop","slug":"incomplete-data-input-length-is","errorCode":null,"errorMessage":"incomplete data, input length is: {}","messagePattern":"incomplete data, input length is: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPuller.java","lineNumber":164,"sourceCode":"    if (null != asideBuffer && asideBuffer.length() > 0) {\n      if (asideBuffer.remaining() > 0) {\n        final byte[] output = asideBuffer.getByteBuffer().array();\n        final int write = Math.min(asideBuffer.remaining(), input.remaining());\n        input.get(output, asideBuffer.position(), write);\n        asideBuffer.position(asideBuffer.position() + write);\n      }\n\n      if (asideBuffer.remaining() == 0) {\n        asideBuffer.position(0);\n      }\n    }\n\n    if (input.remaining() == 0) {\n      return true;\n    }\n\n    if (input.remaining() < KV_HEADER_LENGTH) {\n      throw new IOException(\"incomplete data, input length is: \" + input.remaining());\n    }\n    final int position = input.position();\n    final int keyLength = input.getInt();\n    final int valueLength = input.getInt();\n    input.position(position);\n    final int kvLength = keyLength + valueLength + KV_HEADER_LENGTH;\n    final int remaining = input.remaining();\n\n    if (kvLength > remaining) {\n      if (null == asideBuffer || asideBuffer.capacity() < kvLength) {\n        asideBuffer = new InputBuffer(BufferType.HEAP_BUFFER, kvLength);\n      }\n      asideBuffer.rewind(0, kvLength);\n\n      input.get(asideBuffer.array(), 0, remaining);\n      asideBuffer.position(remaining);\n    }\n    return true;","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPuller.java#L146-L182","documentation":"BufferPuller.moveData() validates each incoming KV chunk: after handling an aside buffer for oversized records, it requires at least the 8-byte KV header (keyLength int + valueLength int) in the remaining input. One to seven leftover bytes means a partial/corrupt record boundary, and it throws IOException('incomplete data, input length is: N'). It signals misalignment between the producer and consumer of the native buffer channel, not a user config error.","triggerScenarios":"The upstream writer and BufferPuller disagree on record framing: serializer length drift between producer/consumer versions, truncated buffer handoff, or corrupted intermediate data on the reduce side of the nativetask pipeline.","commonSituations":"Mixed Hadoop versions between job artifacts and the cluster's nativetask library during rolling upgrades; custom serializers emitting a different header layout than the deserializer expects; memory/hardware corruption of spill files.","solutions":["Verify that all nodes and the submitted job use the same Hadoop/nativetask build (no mixed-version rolling upgrade)","If custom INativeSerializer/Deserializer pairs are in play, check that serialized KV framing (8-byte header + payload) matches on both sides","Re-run the job; persistent occurrences on specific nodes point to a bad native install or hardware - inspect the affected NodeManager"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  boolean done = puller.moveData();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"incomplete data\")) {\n    // framing/truncation between producer and native consumer:\n    // verify version alignment and serializer framing; do not silently retry corrupt data\n    throw new IOException(\"nativetask KV framing error; check jar/so alignment\", e);\n  }\n  throw e;\n}","preventionTips":["Keep Hadoop and nativetask versions uniform across client and cluster during upgrades","Round-trip test custom serializers: emitted bytes must always contain complete 8-byte headers + payload","Treat recurring occurrences on one node as a hardware/native-install investigation"],"tags":["nativetask","buffer-management","data-corruption","internal-api"],"backgroundTag":"truncated-stream-data","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}