{"record":{"id":"7345acbb6e0be852","repo":"apache/hadoop","slug":"buffer-too-small-to-hold-value","errorCode":null,"errorMessage":"Buffer too small to hold value","messagePattern":"Buffer too small to hold value","errorType":"exception","errorClass":"IndexOutOfBoundsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":1877,"sourceCode":"         * large enough to hold the whole value (starting from the offset). The\n         * value part of the key-value pair pointed by the current cursor is not\n         * cached and can only be examined once. Calling any of the following\n         * functions more than once without moving the cursor will result in\n         * exception: {@link #getValue(byte[])}, {@link #getValue(byte[], int)},\n         * {@link #getValueStream}.\n         *\n         * @param buf buf.\n         * @param offset offset.\n         * @return the length of the value. Does not require\n         *         isValueLengthKnown() to be true.\n         * @throws IOException raised on errors performing I/O.\n         */\n        public int getValue(byte[] buf, int offset) throws IOException {\n          DataInputStream dis = getValueStream();\n          try {\n            if (isValueLengthKnown()) {\n              if ((offset | (buf.length - offset - vlen)) < 0) {\n                throw new IndexOutOfBoundsException(\n                    \"Buffer too small to hold value\");\n              }\n              dis.readFully(buf, offset, vlen);\n              return vlen;\n            }\n\n            int nextOffset = offset;\n            while (nextOffset < buf.length) {\n              int n = dis.read(buf, nextOffset, buf.length - nextOffset);\n              if (n < 0) {\n                break;\n              }\n              nextOffset += n;\n            }\n            if (dis.read() >= 0) {\n              // attempt to read one more byte to determine whether we reached\n              // the\n              // end or not.","sourceCodeStart":1859,"sourceCodeEnd":1895,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L1859-L1895","documentation":"IndexOutOfBoundsException from Scanner.Entry.getValue(byte[] buf, int offset) on the known-length path. The same (offset | (buf.length - offset - vlen)) < 0 check rejects a negative offset or a buffer with fewer than vlen bytes of space at offset. Here vlen is authoritative (isValueLengthKnown() is true), so the buffer is provably too small.","triggerScenarios":"Calling getValue(buf, offset) when isValueLengthKnown() is true and buf.length - offset < vlen: fixed 4KB/8KB buffers reading large values, or an offset that eats into the needed space.","commonSituations":"Reusing the key buffer for values, sizing buffers from a config assumption ('values are small') that a large record violates, or tail-offset slicing into a pooled buffer.","solutions":["Allocate from the entry: byte[] v = new byte[entry.getValueLength()] before calling getValue.","Guard: offset >= 0 && offset + entry.getValueLength() <= buf.length, else grow the buffer.","If memory is tight and values can be large, switch to the getValueStream() path and copy in chunks instead of one flat buffer."],"exampleFix":"// before\nbyte[] v = new byte[4096];\nentry.getValue(v, 0); // throws when vlen > 4096\n// after\nif (!entry.isValueLengthKnown() || entry.getValueLength() > v.length) {\n  v = new byte[entry.getValueLength()];\n}\nint n = entry.getValue(v, 0);","handlingStrategy":"validation","validationCode":"if (entry.isValueLengthKnown()) {\n  int vlen = entry.getValueLength();\n  if (offset < 0 || buf.length - offset < vlen) {\n    buf = new byte[vlen]; offset = 0;\n  }\n}\nint n = entry.getValue(buf, offset);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Allocate value buffers from getValueLength() when the length is known.","Never reuse a fixed 4KB/8KB buffer for unbounded values.","Add an integration test with values bigger than your buffer size."],"tags":["tfile","hadoop-common","index-out-of-bounds","buffer","value"],"backgroundTag":"buffer-overflow-guard","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}