apache/hadoop · critical · IOException

We expect to read {}, but we actually read: {}

Error message

We expect to read {}, but we actually read: {}

What it means

After deserializing a whole input buffer, BufferPushee accumulates the byte count each deserializeKV() reports and requires the total to equal the buffer's remaining() captured up front. A mismatch throws IOException('We expect to read R, but we actually read: T') - the deserializer's self-reported lengths do not add up to the bytes actually present, i.e. the data or the length framing is inconsistent. This is the strongest corruption signal of the buffer-handoff checks.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPushee.java:132

  }

  @SuppressWarnings("unchecked")
  private boolean write(InputBuffer input) throws IOException {
    if (closed) {
      return false;
    }
    int totalRead = 0;
    final int remain = input.remaining();
    this.nativeReader.reset(input);
    while (remain > totalRead) {
      final int read = deserializer.deserializeKV(nativeReader, tmpOutputKey, tmpOutputValue);
      if (read != 0) {
        totalRead += read;
        writer.write((OK) (tmpOutputKey.v), (OV) (tmpOutputValue.v));
      }
    }
    if (remain != totalRead) {
      throw new IOException("We expect to read " + remain +
                            ", but we actually read: " + totalRead);
    }
    return true;
  }

  @Override
  public void close() throws IOException {
    if (closed) {
      return;
    }
    if (null != writer) {
      writer.close(null);
    }
    if (null != nativeReader) {
      nativeReader.close();
    }
    closed = true;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Unit-test the custom serializer/deserializer pair against round-trip framing: sum of reported reads must equal serialized size for random KV sequences
2. Align Hadoop jar and libnativetask.so versions across client and cluster
3. Run the job with the native collector disabled to confirm the data path itself is sound, isolating the native framing layer

try {
  boolean done = pushee.collectData();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("We expect to read")) {
    // deserializer-reported byte counts disagree with buffer contents -> data/framing corruption
    // capture the buffer snapshot for diagnosis; rerun the task once on a different node
    throw new IOException("nativetask length mismatch; suspected corruption or serializer bug", e);
  }
  throw e;
}

When it happens

Trigger: deserializeKV reports per-record read sizes that under- or over-count versus the buffer contents: mismatched serializer/deserializer implementations, records whose declared key/value lengths disagree with the payload, or buffers corrupted between producer and consumer.

Common situations: Custom INativeSerializer getBytesWritten/value length bugs; cross-version clusters where framing changed; disk or network corruption of intermediate data; native library built with different struct packing/alignment than the Java side assumes.

Related errors

• incomplete data, input length is: {}
• incomplete data, input length is: {}
• encoded array component type {} is not a candidate primitive
• encoded array length is negative {}
• Unexpected block size: ${numBytes}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1b7460d5e13333fd. Report an issue: GitHub.