apache/hadoop · error · IOException

negative length keys not allowed: {}

Error message

negative length keys not allowed: {}

What it means

After serialize(key) fills the in-memory buffer, append reads buffer.getLength() into an int; if the serialized key was larger than Integer.MAX_VALUE bytes the length wraps negative and append throws 'negative length keys not allowed'. So in practice this error means an oversized (>2GB) serialized key, or a custom serializer/Writable whose write() emits a bogus length. Sequence files are not designed for such keys.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1485

     * @throws IOException raised on errors performing I/O.
     */
    @SuppressWarnings("unchecked")
    public synchronized void append(Object key, Object val)
      throws IOException {
      if (key.getClass() != keyClass)
        throw new IOException("wrong key class: "+key.getClass().getName()
                              +" is not "+keyClass);
      if (val.getClass() != valClass)
        throw new IOException("wrong value class: "+val.getClass().getName()
                              +" is not "+valClass);

      buffer.reset();

      // Append the 'key'
      keySerializer.serialize(key);
      int keyLength = buffer.getLength();
      if (keyLength < 0)
        throw new IOException("negative length keys not allowed: " + key);

      // Append the 'value'
      if (compress == CompressionType.RECORD) {
        deflateFilter.resetState();
        compressedValSerializer.serialize(val);
        deflateOut.flush();
        deflateFilter.finish();
      } else {
        uncompressedValSerializer.serialize(val);
      }

      // Write the record out
      checkAndWriteSync();                                // sync
      out.writeInt(buffer.getLength());                   // total record length
      out.writeInt(keyLength);                            // key portion length
      out.write(buffer.getData(), 0, buffer.getLength()); // data
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Move the bulk data into the value and keep keys small (an ID, hash, or offset)
2. Cap key size in application code: serialize to a scratch DataOutputBuffer first and reject lengths above a sane threshold
3. Fix the custom Writable/Serializer so write() always emits a correct, bounded length

Example fix

// before
MyHugeKey key = new MyHugeKey(entireDataset); // serializes to >2GB
w.append(key, val); // negative length keys not allowed

// after
DataOutputBuffer probe = new DataOutputBuffer();
newKey.write(probe);
if (probe.getLength() > 64 * 1024 * 1024) {
  throw new IllegalArgumentException("key too large: " + probe.getLength());
}
w.append(newKey, val);

DataOutputBuffer probe = new DataOutputBuffer();
((Writable) key).write(probe);
if (probe.getLength() < 0 || probe.getLength() > MAX_KEY_BYTES) {
  throw new IllegalArgumentException("serialized key too large: " + probe.getLength());
}

try {
  w.append(key, val);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("negative length keys")) {
    // key serialization exceeded 2GB (or serializer is broken): shrink the key, never retry as-is
  } else { throw e; }
}

When it happens

Trigger: Putting the whole payload into the key (e.g. entire file/graph in the Writable key) so its serialized form exceeds 2^31-1 bytes; a hand-written Writable.write() or Serializer that writes a corrupt huge/negative length; a corrupted reused key object carrying bogus state.

Common situations: Modeling mistakes where the 'key' is the document; tests serializing oversized synthetic buffers; buggy custom serialization after a format change; silent int overflow in size-computing helper code.

Related errors

• Key/value class provided does not match the file
• A record version mismatch occurred. Expecting v{}, found v{}
• Compression option provided does not match the file
• wrong key class: {} is not {}
• wrong value class: {} is not {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f04e9e7cc73fff76. Report an issue: GitHub.