{"record":{"id":"16ec576451cc4e17","repo":"apache/hadoop","slug":"buffer-not-enough-to-store-the-key","errorCode":null,"errorMessage":"Buffer not enough to store the key","messagePattern":"Buffer not enough to store the key","errorType":"exception","errorClass":"IndexOutOfBoundsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":1806,"sourceCode":"        public int getKey(byte[] buf) throws IOException {\n          return getKey(buf, 0);\n        }\n\n        /**\n         * Copy the key into user supplied buffer.\n         * \n         * @param buf\n         *          The buffer supplied by user.\n         * @param offset\n         *          The starting offset of the user buffer where we should copy\n         *          the key into. Requiring the key-length + offset no greater\n         *          than the buffer length.\n         * @return The length of the key.\n         * @throws IOException raised on errors performing I/O.\n         */\n        public int getKey(byte[] buf, int offset) throws IOException {\n          if ((offset | (buf.length - offset - klen)) < 0) {\n            throw new IndexOutOfBoundsException(\n                \"Buffer not enough to store the key\");\n          }\n          System.arraycopy(keyBuffer, 0, buf, offset, klen);\n          return klen;\n        }\n\n        /**\n         * Streaming access to the key. Useful for desrializing the key into\n         * user objects.\n         * \n         * @return The input stream.\n         */\n        public DataInputStream getKeyStream() {\n          keyDataInputStream.reset(keyBuffer, klen);\n          return keyDataInputStream;\n        }\n\n        /**","sourceCodeStart":1788,"sourceCodeEnd":1824,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L1788-L1824","documentation":"IndexOutOfBoundsException thrown by Scanner.Entry.getKey(byte[] buf, int offset). The bit-trick check (offset | (buf.length - offset - klen)) < 0 rejects any negative offset or any offset where the remaining space (buf.length - offset) is smaller than the already-parsed key length klen. The key length is known before you copy, so this is always a caller sizing bug, not file corruption.","triggerScenarios":"Calling entry.getKey(buf, offset) with buf smaller than entry.getKeyLength(), with a negative offset, or with an offset that leaves fewer than klen bytes remaining in buf (e.g. offset near the tail of a reused buffer).","commonSituations":"Reusing one fixed-size scratch buffer for keys of varying length, slicing a buffer with a tail offset, or assuming all keys share a maximum size (TFile permits keys up to 64KB).","solutions":["Size the buffer from the entry itself: use entry.getKeyLength() before calling getKey(buf, offset).","Guard the call: require offset >= 0 && offset + entry.getKeyLength() <= buf.length.","If you only need the bytes, allocate exactly: byte[] key = new byte[entry.getKeyLength()]; entry.getKey(key, 0)."],"exampleFix":"// before\nbyte[] buf = new byte[64];\nint klen = entry.getKey(buf, 0); // throws when key > 64 bytes\n// after\nif (buf.length < entry.getKeyLength()) buf = new byte[entry.getKeyLength()];\nint klen = entry.getKey(buf, 0);","handlingStrategy":"validation","validationCode":"int klen = entry.getKeyLength();\nif (offset < 0 || buf.length - offset < klen) {\n  buf = new byte[Math.max(buf.length, klen)]; offset = 0; // or fail fast\n}\nint copied = entry.getKey(buf, offset);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always size from entry.getKeyLength(), never from an assumed constant.","Keep offset math in one place; assert offset >= 0 in debug builds.","For variable-size keys prefer allocating per key: new byte[entry.getKeyLength()]."],"tags":["tfile","hadoop-common","index-out-of-bounds","buffer","key"],"backgroundTag":"buffer-overflow-guard","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}