apache/hadoop · error · IndexOutOfBoundsException

Buffer not enough to store the key

Error message

Buffer not enough to store the key

What it means

IndexOutOfBoundsException thrown by Scanner.Entry.getKey(byte[] buf, int offset). The bit-trick check (offset | (buf.length - offset - klen)) < 0 rejects any negative offset or any offset where the remaining space (buf.length - offset) is smaller than the already-parsed key length klen. The key length is known before you copy, so this is always a caller sizing bug, not file corruption.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:1806

        public int getKey(byte[] buf) throws IOException {
          return getKey(buf, 0);
        }

        /**
         * Copy the key into user supplied buffer.
         * 
         * @param buf
         *          The buffer supplied by user.
         * @param offset
         *          The starting offset of the user buffer where we should copy
         *          the key into. Requiring the key-length + offset no greater
         *          than the buffer length.
         * @return The length of the key.
         * @throws IOException raised on errors performing I/O.
         */
        public int getKey(byte[] buf, int offset) throws IOException {
          if ((offset | (buf.length - offset - klen)) < 0) {
            throw new IndexOutOfBoundsException(
                "Buffer not enough to store the key");
          }
          System.arraycopy(keyBuffer, 0, buf, offset, klen);
          return klen;
        }

        /**
         * Streaming access to the key. Useful for desrializing the key into
         * user objects.
         * 
         * @return The input stream.
         */
        public DataInputStream getKeyStream() {
          keyDataInputStream.reset(keyBuffer, klen);
          return keyDataInputStream;
        }

        /**

View on GitHub (pinned to 2add963021)

Solutions

1. Size the buffer from the entry itself: use entry.getKeyLength() before calling getKey(buf, offset).
2. Guard the call: require offset >= 0 && offset + entry.getKeyLength() <= buf.length.
3. If you only need the bytes, allocate exactly: byte[] key = new byte[entry.getKeyLength()]; entry.getKey(key, 0).

Example fix

// before
byte[] buf = new byte[64];
int klen = entry.getKey(buf, 0); // throws when key > 64 bytes
// after
if (buf.length < entry.getKeyLength()) buf = new byte[entry.getKeyLength()];
int klen = entry.getKey(buf, 0);

int klen = entry.getKeyLength();
if (offset < 0 || buf.length - offset < klen) {
  buf = new byte[Math.max(buf.length, klen)]; offset = 0; // or fail fast
}
int copied = entry.getKey(buf, offset);

When it happens

Trigger: Calling entry.getKey(buf, offset) with buf smaller than entry.getKeyLength(), with a negative offset, or with an offset that leaves fewer than klen bytes remaining in buf (e.g. offset near the tail of a reused buffer).

Common situations: Reusing one fixed-size scratch buffer for keys of varying length, slicing a buffer with a tail offset, or assuming all keys share a maximum size (TFile permits keys up to 64KB).

Related errors

• Buffer too small to hold value
• offset: %s is out of range [%s, %s]
• Requested more bytes than destination buffer size: request l
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/16ec576451cc4e17. Report an issue: GitHub.