apache/hadoop · error · IOException

Negative key-length not allowed: {keyLength} for {key}

Error message

Negative key-length not allowed: {keyLength} for {key}

What it means

InMemoryWriter.append(DataInputBuffer, DataInputBuffer) computes keyLength as key.getLength() - key.getPosition(); a negative value means the buffer's position is past its length, i.e. an already-consumed or misused buffer. Writing it would emit a bogus IFile record length, so the append fails fast with IOException.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryWriter.java:52

public class InMemoryWriter<K, V> extends Writer<K, V> {
  private DataOutputStream out;
  
  public InMemoryWriter(BoundedByteArrayOutputStream arrayStream) {
    super(null);
    this.out = 
      new DataOutputStream(new IFileOutputStream(arrayStream));
  }
  
  public void append(K key, V value) throws IOException {
    throw new UnsupportedOperationException
    ("InMemoryWriter.append(K key, V value");
  }
  
  public void append(DataInputBuffer key, DataInputBuffer value)
  throws IOException {
    int keyLength = key.getLength() - key.getPosition();
    if (keyLength < 0) {
      throw new IOException("Negative key-length not allowed: " + keyLength + 
                            " for " + key);
    }
    
    int valueLength = value.getLength() - value.getPosition();
    if (valueLength < 0) {
      throw new IOException("Negative value-length not allowed: " + 
                            valueLength + " for " + value);
    }

    WritableUtils.writeVInt(out, keyLength);
    WritableUtils.writeVInt(out, valueLength);
    out.write(key.getData(), key.getPosition(), keyLength); 
    out.write(value.getData(), value.getPosition(), valueLength); 
  }

  public void close() throws IOException {
    // Write EOF_MARKER for key/value length
    WritableUtils.writeVInt(out, IFile.EOF_MARKER);

View on GitHub (pinned to 2add963021)

Solutions

1. reset() the buffer to its data before appending: key.reset(keyBytes, 0, keyBytes.length).
2. Pass freshly populated buffers to append() instead of reusing consumed ones.
3. If this fires inside framework code (no custom writer usage), capture the stack and map id and report it upstream.

Example fix

// before: buffer was advanced by a previous read, position > length
writer.append(consumedKeyBuf, valueBuf);
// after: re-bind the buffer before appending
consumedKeyBuf.reset(keyBytes, 0, keyBytes.length);
writer.append(consumedKeyBuf, valueBuf);

// Guard before InMemoryWriter.append(DataInputBuffer, DataInputBuffer)
static boolean isAppendable(org.apache.hadoop.io.DataInputBuffer b) {
  return b != null && b.getLength() - b.getPosition() >= 0;
}
if (!isAppendable(keyBuf)) { keyBuf.reset(keyBytes, 0, keyBytes.length); }

When it happens

Trigger: Calling append with a DataInputBuffer that a previous read advanced without re-binding it via reset(data, offset, len); the mem-to-mem merger passing buffers whose position exceeds their valid length.

Common situations: Code that reuses one DataInputBuffer across iterations of a reader and passes it onward without reset; rarely, framework IntermediateMemoryToMemoryMerger bugs on specific versions.

Related errors

• Negative value-length not allowed: {valueLength} for {value}
• Rec# {recNo}: Failed to skip past key of length: {currentKey
• Rec# {recNo}: Failed to skip past value of length: {currentV
• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ec2366093acbbabe. Report an issue: GitHub.