apache/hadoop · error · IOException

output buffer not set

Error message

output buffer not set

What it means

BufferPullee (reduce-side native buffer consumer) obtains its output buffer from target.getOutputBuffer() at construction; load() refuses to run when that buffer is null and throws IOException('output buffer not set'). It means the downstream target handler never initialized its native output buffer before a pull was attempted - an internal wiring/initialization-order defect in the nativetask data path rather than a user configuration issue.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/handlers/BufferPullee.java:70

    this.rIter = rIter;
    tmpInputKey = new SizedWritable<IK>(iKClass);
    tmpInputValue = new SizedWritable<IV>(iVClass);

    if (null != iKClass && null != iVClass) {
      this.serializer = new KVSerializer<IK, IV>(iKClass, iVClass);
    }
    this.outputBuffer = target.getOutputBuffer();
    this.target = target;
  }

  @Override
  public int load() throws IOException {
    if (closed) {
      return 0;
    }
    
    if (null == outputBuffer) {
      throw new IOException("output buffer not set");
    }

    this.nativeWriter = new ByteBufferDataWriter(target);
    outputBuffer.rewind();

    int written = 0;
    boolean firstKV = true;

    if (inputKVBufferd) {
      written += serializer.serializeKV(nativeWriter, tmpInputKey, tmpInputValue);
      inputKVBufferd = false;
      firstKV = false;
    }

    while (rIter.next()) {
      inputKVBufferd = false;
      tmpInputKey.readFields(rIter.getKey());
      tmpInputValue.readFields(rIter.getValue());

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure the target handler is fully initialized (native object created, init(conf) run) before constructing or using the BufferPullee
2. Null-check target.getOutputBuffer() during wiring and fail with context on which handler lacks a buffer
3. If using stock handlers only, check for jar/so version mismatch and realign the installation

Example fix

// before
BufferPullee pullee = new BufferPullee(target);
pullee.load(); // IOException if target had no output buffer

// after
if (target.getOutputBuffer() == null) {
  throw new IOException("target " + target.getClass().getName() + " has no output buffer; init it first");
}
pullee.load();

if (target.getOutputBuffer() == null) {
  throw new IllegalStateException(
      "target " + target.getClass().getSimpleName() + " not initialized: no output buffer");
}
pullee.load();

try {
  int written = pullee.load();
} catch (IOException e) {
  if ("output buffer not set".equals(e.getMessage())) {
    // wiring bug: re-init target handler and rebuild the pullee before retrying
  }
  throw e;
}

When it happens

Trigger: Constructing a BufferPullee against a target whose native handler was not initialized (init not called, or createNativeObject failed silently earlier), then calling load(); also after close() interplay if the target was rebuilt without re-acquiring the buffer.

Common situations: Developing custom nativetask platforms/handlers and wiring pull/push pipelines by hand; version drift where the target handler implementation stopped publishing an output buffer.

Related errors

• incomplete data, input length is: {}
• incomplete data, input length is: {}
• Native output collector cannot be loaded;
• We expect to read {}, but we actually read: {}
• FileMetadata not match key:" + key

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a672f3c6a66955fb. Report an issue: GitHub.