apache/hadoop · error · IOException

incomplete data, input length is: {}

Error message

incomplete data, input length is: {}

What it means

BufferPushee.collectData() applies the same framing contract on the push side: it flushes a pending aside buffer, returns early when input is empty, and requires the remaining bytes to contain at least the KV_HEADER_LENGTH (8 bytes). A positive remainder below that throws IOException('incomplete data, input length is: N') - the input buffer handed to the native pusher ends mid-header.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPushee.java:93

        final byte[] output = asideBuffer.getByteBuffer().array();
        final int write = Math.min(asideBuffer.remaining(), input.remaining());
        input.get(output, asideBuffer.position(), write);
        asideBuffer.position(asideBuffer.position() + write);
      }

      if (asideBuffer.remaining() == 0 && asideBuffer.position() > 0) {
        asideBuffer.position(0);
        write(asideBuffer);
        asideBuffer.rewind(0, 0);
      }
    }

    if (input.remaining() == 0) {
      return true;
    }

    if (input.remaining() < KV_HEADER_LENGTH) {
      throw new IOException("incomplete data, input length is: " + input.remaining());
    }
    final int position = input.position();
    final int keyLength = input.getInt();
    final int valueLength = input.getInt();
    input.position(position);
    final int kvLength = keyLength + valueLength + KV_HEADER_LENGTH;
    final int remaining = input.remaining();

    if (kvLength > remaining) {
      if (null == asideBuffer || asideBuffer.capacity() < kvLength) {
        asideBuffer = new InputBuffer(BufferType.HEAP_BUFFER, kvLength);
      }
      asideBuffer.rewind(0, kvLength);

      input.get(asideBuffer.array(), 0, remaining);
      asideBuffer.position(remaining);
    } else {
      write(buffer);

View on GitHub (pinned to 2add963021)

Solutions

1. Align versions of jars and libnativetask.so across the cluster and the job submission
2. For custom serializers, verify every record write emits exactly keyLen int + valueLen int + payload and that reported lengths match bytes written
3. Revert buffer-size overrides (native status/spill buffer settings) to defaults to rule out mis-tuned boundaries

try {
  boolean done = pushee.collectData();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("incomplete data")) {
    // map-side buffer ends mid-header: verify serializer length accounting and version alignment
    throw new IOException("nativetask push framing error; check serializer lengths", e);
  }
  throw e;
}

When it happens

Trigger: The map-side serializer writes a KV record whose header is split across the buffer boundary in a way the aside-buffer logic cannot accommodate, or the input buffer passed into collectData() was truncated/misaligned by upstream code.

Common situations: Custom serialization implementations that miscalculate lengths; buffer size tuning (io sort / native buffer sizes) exposing boundary bugs; version mismatch between the Java-side writer and native reader during upgrades.

Related errors

• incomplete data, input length is: {}
• output buffer not set
• We expect to read {}, but we actually read: {}
• FileMetadata not match key:" + key
• no such method

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ff316a5e66d28108. Report an issue: GitHub.