apache/hadoop · error · IndexOutOfBoundsException

Buffer too small to hold value

Error message

Buffer too small to hold value

What it means

IndexOutOfBoundsException from Scanner.Entry.getValue(byte[] buf, int offset) on the known-length path. The same (offset | (buf.length - offset - vlen)) < 0 check rejects a negative offset or a buffer with fewer than vlen bytes of space at offset. Here vlen is authoritative (isValueLengthKnown() is true), so the buffer is provably too small.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:1877

         * large enough to hold the whole value (starting from the offset). The
         * value part of the key-value pair pointed by the current cursor is not
         * cached and can only be examined once. Calling any of the following
         * functions more than once without moving the cursor will result in
         * exception: {@link #getValue(byte[])}, {@link #getValue(byte[], int)},
         * {@link #getValueStream}.
         *
         * @param buf buf.
         * @param offset offset.
         * @return the length of the value. Does not require
         *         isValueLengthKnown() to be true.
         * @throws IOException raised on errors performing I/O.
         */
        public int getValue(byte[] buf, int offset) throws IOException {
          DataInputStream dis = getValueStream();
          try {
            if (isValueLengthKnown()) {
              if ((offset | (buf.length - offset - vlen)) < 0) {
                throw new IndexOutOfBoundsException(
                    "Buffer too small to hold value");
              }
              dis.readFully(buf, offset, vlen);
              return vlen;
            }

            int nextOffset = offset;
            while (nextOffset < buf.length) {
              int n = dis.read(buf, nextOffset, buf.length - nextOffset);
              if (n < 0) {
                break;
              }
              nextOffset += n;
            }
            if (dis.read() >= 0) {
              // attempt to read one more byte to determine whether we reached
              // the
              // end or not.

View on GitHub (pinned to 2add963021)

Solutions

1. Allocate from the entry: byte[] v = new byte[entry.getValueLength()] before calling getValue.
2. Guard: offset >= 0 && offset + entry.getValueLength() <= buf.length, else grow the buffer.
3. If memory is tight and values can be large, switch to the getValueStream() path and copy in chunks instead of one flat buffer.

Example fix

// before
byte[] v = new byte[4096];
entry.getValue(v, 0); // throws when vlen > 4096
// after
if (!entry.isValueLengthKnown() || entry.getValueLength() > v.length) {
  v = new byte[entry.getValueLength()];
}
int n = entry.getValue(v, 0);

if (entry.isValueLengthKnown()) {
  int vlen = entry.getValueLength();
  if (offset < 0 || buf.length - offset < vlen) {
    buf = new byte[vlen]; offset = 0;
  }
}
int n = entry.getValue(buf, offset);

When it happens

Trigger: Calling getValue(buf, offset) when isValueLengthKnown() is true and buf.length - offset < vlen: fixed 4KB/8KB buffers reading large values, or an offset that eats into the needed space.

Common situations: Reusing the key buffer for values, sizing buffers from a config assumption ('values are small') that a large record violates, or tail-offset slicing into a pooled buffer.

Related errors

• Buffer not enough to store the key
• offset: %s is out of range [%s, %s]
• Requested more bytes than destination buffer size: request l
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7345acbb6e0be852. Report an issue: GitHub.