{"record":{"id":"ec2366093acbbabe","repo":"apache/hadoop","slug":"negative-key-length-not-allowed-keylength-for","errorCode":null,"errorMessage":"Negative key-length not allowed: {keyLength} for {key}","messagePattern":"Negative key-length not allowed: (.+?) for (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryWriter.java","lineNumber":52,"sourceCode":"public class InMemoryWriter<K, V> extends Writer<K, V> {\n  private DataOutputStream out;\n  \n  public InMemoryWriter(BoundedByteArrayOutputStream arrayStream) {\n    super(null);\n    this.out = \n      new DataOutputStream(new IFileOutputStream(arrayStream));\n  }\n  \n  public void append(K key, V value) throws IOException {\n    throw new UnsupportedOperationException\n    (\"InMemoryWriter.append(K key, V value\");\n  }\n  \n  public void append(DataInputBuffer key, DataInputBuffer value)\n  throws IOException {\n    int keyLength = key.getLength() - key.getPosition();\n    if (keyLength < 0) {\n      throw new IOException(\"Negative key-length not allowed: \" + keyLength + \n                            \" for \" + key);\n    }\n    \n    int valueLength = value.getLength() - value.getPosition();\n    if (valueLength < 0) {\n      throw new IOException(\"Negative value-length not allowed: \" + \n                            valueLength + \" for \" + value);\n    }\n\n    WritableUtils.writeVInt(out, keyLength);\n    WritableUtils.writeVInt(out, valueLength);\n    out.write(key.getData(), key.getPosition(), keyLength); \n    out.write(value.getData(), value.getPosition(), valueLength); \n  }\n\n  public void close() throws IOException {\n    // Write EOF_MARKER for key/value length\n    WritableUtils.writeVInt(out, IFile.EOF_MARKER);","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryWriter.java#L34-L70","documentation":"InMemoryWriter.append(DataInputBuffer, DataInputBuffer) computes keyLength as key.getLength() - key.getPosition(); a negative value means the buffer's position is past its length, i.e. an already-consumed or misused buffer. Writing it would emit a bogus IFile record length, so the append fails fast with IOException.","triggerScenarios":"Calling append with a DataInputBuffer that a previous read advanced without re-binding it via reset(data, offset, len); the mem-to-mem merger passing buffers whose position exceeds their valid length.","commonSituations":"Code that reuses one DataInputBuffer across iterations of a reader and passes it onward without reset; rarely, framework IntermediateMemoryToMemoryMerger bugs on specific versions.","solutions":["reset() the buffer to its data before appending: key.reset(keyBytes, 0, keyBytes.length).","Pass freshly populated buffers to append() instead of reusing consumed ones.","If this fires inside framework code (no custom writer usage), capture the stack and map id and report it upstream."],"exampleFix":"// before: buffer was advanced by a previous read, position > length\nwriter.append(consumedKeyBuf, valueBuf);\n// after: re-bind the buffer before appending\nconsumedKeyBuf.reset(keyBytes, 0, keyBytes.length);\nwriter.append(consumedKeyBuf, valueBuf);","handlingStrategy":"validation","validationCode":"// Guard before InMemoryWriter.append(DataInputBuffer, DataInputBuffer)\nstatic boolean isAppendable(org.apache.hadoop.io.DataInputBuffer b) {\n  return b != null && b.getLength() - b.getPosition() >= 0;\n}\nif (!isAppendable(keyBuf)) { keyBuf.reset(keyBytes, 0, keyBytes.length); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always reset(data, offset, length) a DataInputBuffer before passing it to a writer.","Never share one advanced buffer between a reader loop and a writer without rebinding."],"tags":["hadoop","mapreduce","shuffle","ifile","buffer","validation"],"backgroundTag":"invalid-buffer-state","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}