{"record":{"id":"ff316a5e66d28108","repo":"apache/hadoop","slug":"incomplete-data-input-length-is-ff316a","errorCode":null,"errorMessage":"incomplete data, input length is: {}","messagePattern":"incomplete data, input length is: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPushee.java","lineNumber":93,"sourceCode":"        final byte[] output = asideBuffer.getByteBuffer().array();\n        final int write = Math.min(asideBuffer.remaining(), input.remaining());\n        input.get(output, asideBuffer.position(), write);\n        asideBuffer.position(asideBuffer.position() + write);\n      }\n\n      if (asideBuffer.remaining() == 0 && asideBuffer.position() > 0) {\n        asideBuffer.position(0);\n        write(asideBuffer);\n        asideBuffer.rewind(0, 0);\n      }\n    }\n\n    if (input.remaining() == 0) {\n      return true;\n    }\n\n    if (input.remaining() < KV_HEADER_LENGTH) {\n      throw new IOException(\"incomplete data, input length is: \" + input.remaining());\n    }\n    final int position = input.position();\n    final int keyLength = input.getInt();\n    final int valueLength = input.getInt();\n    input.position(position);\n    final int kvLength = keyLength + valueLength + KV_HEADER_LENGTH;\n    final int remaining = input.remaining();\n\n    if (kvLength > remaining) {\n      if (null == asideBuffer || asideBuffer.capacity() < kvLength) {\n        asideBuffer = new InputBuffer(BufferType.HEAP_BUFFER, kvLength);\n      }\n      asideBuffer.rewind(0, kvLength);\n\n      input.get(asideBuffer.array(), 0, remaining);\n      asideBuffer.position(remaining);\n    } else {\n      write(buffer);","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPushee.java#L75-L111","documentation":"BufferPushee.collectData() applies the same framing contract on the push side: it flushes a pending aside buffer, returns early when input is empty, and requires the remaining bytes to contain at least the KV_HEADER_LENGTH (8 bytes). A positive remainder below that throws IOException('incomplete data, input length is: N') - the input buffer handed to the native pusher ends mid-header.","triggerScenarios":"The map-side serializer writes a KV record whose header is split across the buffer boundary in a way the aside-buffer logic cannot accommodate, or the input buffer passed into collectData() was truncated/misaligned by upstream code.","commonSituations":"Custom serialization implementations that miscalculate lengths; buffer size tuning (io sort / native buffer sizes) exposing boundary bugs; version mismatch between the Java-side writer and native reader during upgrades.","solutions":["Align versions of jars and libnativetask.so across the cluster and the job submission","For custom serializers, verify every record write emits exactly keyLen int + valueLen int + payload and that reported lengths match bytes written","Revert buffer-size overrides (native status/spill buffer settings) to defaults to rule out mis-tuned boundaries"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  boolean done = pushee.collectData();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"incomplete data\")) {\n    // map-side buffer ends mid-header: verify serializer length accounting and version alignment\n    throw new IOException(\"nativetask push framing error; check serializer lengths\", e);\n  }\n  throw e;\n}","preventionTips":["For custom serializers, assert keyLen+valueLen+8 accounting matches bytes written in unit tests","Avoid ad-hoc native buffer-size tuning; keep spill/buffer settings at validated defaults","Pin one Hadoop build across the pipeline to avoid framing drift"],"tags":["nativetask","buffer-management","data-corruption","internal-api"],"backgroundTag":"truncated-stream-data","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}