apache/hadoop · error · IOException

Negative value-length not allowed: {valueLength} for {value}

Error message

Negative value-length not allowed: {valueLength} for {value}

What it means

InMemoryWriter.append(DataInputBuffer, DataInputBuffer) computes valueLength as value.getLength() - value.getPosition(); a negative value means the value buffer's position is past its length, so the record would carry a nonsense length. The append fails fast instead of writing corrupt IFile output.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryWriter.java:58

      new DataOutputStream(new IFileOutputStream(arrayStream));
  }
  
  public void append(K key, V value) throws IOException {
    throw new UnsupportedOperationException
    ("InMemoryWriter.append(K key, V value");
  }
  
  public void append(DataInputBuffer key, DataInputBuffer value)
  throws IOException {
    int keyLength = key.getLength() - key.getPosition();
    if (keyLength < 0) {
      throw new IOException("Negative key-length not allowed: " + keyLength + 
                            " for " + key);
    }
    
    int valueLength = value.getLength() - value.getPosition();
    if (valueLength < 0) {
      throw new IOException("Negative value-length not allowed: " + 
                            valueLength + " for " + value);
    }

    WritableUtils.writeVInt(out, keyLength);
    WritableUtils.writeVInt(out, valueLength);
    out.write(key.getData(), key.getPosition(), keyLength); 
    out.write(value.getData(), value.getPosition(), valueLength); 
  }

  public void close() throws IOException {
    // Write EOF_MARKER for key/value length
    WritableUtils.writeVInt(out, IFile.EOF_MARKER);
    WritableUtils.writeVInt(out, IFile.EOF_MARKER);
    
    // Close the stream 
    out.close();
    out = null;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. reset() the value buffer to its data before appending: value.reset(valueBytes, 0, valueBytes.length).
2. Allocate or rebind fresh buffers per append instead of passing consumed ones.
3. For framework-internal occurrences, capture the stack trace and report upstream.

Example fix

// before: value buffer position advanced past its data
writer.append(keyBuf, consumedValueBuf);
// after: re-bind before appending
consumedValueBuf.reset(valueBytes, 0, valueBytes.length);
writer.append(keyBuf, consumedValueBuf);

// Guard the value buffer before append
if (valueBuf.getLength() - valueBuf.getPosition() < 0) {
  valueBuf.reset(valueBytes, 0, valueBytes.length);
}

When it happens

Trigger: Calling append with a value DataInputBuffer that was already consumed or never reset; reader loops handing the same buffer to multiple writers after advancing it.

Common situations: Buffer-reuse patterns in custom merge/copy code; framework-internal occurrences are rare and version-specific.

Related errors

• Negative key-length not allowed: {keyLength} for {key}
• Rec# {recNo}: Failed to skip past key of length: {currentKey
• Rec# {recNo}: Failed to skip past value of length: {currentV
• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5ed9bde7d4811a9f. Report an issue: GitHub.