{"record":{"id":"f04e9e7cc73fff76","repo":"apache/hadoop","slug":"negative-length-keys-not-allowed","errorCode":null,"errorMessage":"negative length keys not allowed: {}","messagePattern":"negative length keys not allowed: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java","lineNumber":1485,"sourceCode":"     * @throws IOException raised on errors performing I/O.\n     */\n    @SuppressWarnings(\"unchecked\")\n    public synchronized void append(Object key, Object val)\n      throws IOException {\n      if (key.getClass() != keyClass)\n        throw new IOException(\"wrong key class: \"+key.getClass().getName()\n                              +\" is not \"+keyClass);\n      if (val.getClass() != valClass)\n        throw new IOException(\"wrong value class: \"+val.getClass().getName()\n                              +\" is not \"+valClass);\n\n      buffer.reset();\n\n      // Append the 'key'\n      keySerializer.serialize(key);\n      int keyLength = buffer.getLength();\n      if (keyLength < 0)\n        throw new IOException(\"negative length keys not allowed: \" + key);\n\n      // Append the 'value'\n      if (compress == CompressionType.RECORD) {\n        deflateFilter.resetState();\n        compressedValSerializer.serialize(val);\n        deflateOut.flush();\n        deflateFilter.finish();\n      } else {\n        uncompressedValSerializer.serialize(val);\n      }\n\n      // Write the record out\n      checkAndWriteSync();                                // sync\n      out.writeInt(buffer.getLength());                   // total record length\n      out.writeInt(keyLength);                            // key portion length\n      out.write(buffer.getData(), 0, buffer.getLength()); // data\n    }\n","sourceCodeStart":1467,"sourceCodeEnd":1503,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L1467-L1503","documentation":"After serialize(key) fills the in-memory buffer, append reads buffer.getLength() into an int; if the serialized key was larger than Integer.MAX_VALUE bytes the length wraps negative and append throws 'negative length keys not allowed'. So in practice this error means an oversized (>2GB) serialized key, or a custom serializer/Writable whose write() emits a bogus length. Sequence files are not designed for such keys.","triggerScenarios":"Putting the whole payload into the key (e.g. entire file/graph in the Writable key) so its serialized form exceeds 2^31-1 bytes; a hand-written Writable.write() or Serializer that writes a corrupt huge/negative length; a corrupted reused key object carrying bogus state.","commonSituations":"Modeling mistakes where the 'key' is the document; tests serializing oversized synthetic buffers; buggy custom serialization after a format change; silent int overflow in size-computing helper code.","solutions":["Move the bulk data into the value and keep keys small (an ID, hash, or offset)","Cap key size in application code: serialize to a scratch DataOutputBuffer first and reject lengths above a sane threshold","Fix the custom Writable/Serializer so write() always emits a correct, bounded length"],"exampleFix":"// before\nMyHugeKey key = new MyHugeKey(entireDataset); // serializes to >2GB\nw.append(key, val); // negative length keys not allowed\n\n// after\nDataOutputBuffer probe = new DataOutputBuffer();\nnewKey.write(probe);\nif (probe.getLength() > 64 * 1024 * 1024) {\n  throw new IllegalArgumentException(\"key too large: \" + probe.getLength());\n}\nw.append(newKey, val);","handlingStrategy":"validation","validationCode":"DataOutputBuffer probe = new DataOutputBuffer();\n((Writable) key).write(probe);\nif (probe.getLength() < 0 || probe.getLength() > MAX_KEY_BYTES) {\n  throw new IllegalArgumentException(\"serialized key too large: \" + probe.getLength());\n}","typeGuard":null,"tryCatchPattern":"try {\n  w.append(key, val);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"negative length keys\")) {\n    // key serialization exceeded 2GB (or serializer is broken): shrink the key, never retry as-is\n  } else { throw e; }\n}","preventionTips":["Design keys to be small identifiers; keep payloads in values","Enforce a key-size cap in your record-building code with a serialized-length check"],"tags":["hadoop","sequence-file","overflow","append"],"backgroundTag":"serialized-key-too-large","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}