apache/hadoop · error · IndexOutOfBoundsException

Bad key buffer offset-length combination.

Error message

Bad key buffer offset-length combination.

What it means

Thrown by TFile.Writer.append(byte[] key, int koff, int klen, byte[] value, int voff, int vlen) when the key offset/length pair is invalid: koff or klen negative, koff+klen overflows int, or koff+klen exceeds key.length. The check is the classic bitwise trick (koff | klen | (koff+klen) | (key.length-(koff+klen))) < 0. It fires as IndexOutOfBoundsException before any bytes are written, so the TFile is not corrupted by the bad call.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:396

     *          offset in key buffer.
     * @param klen
     *          length of key.
     * @param value
     *          buffer for value.
     * @param voff
     *          offset in value buffer.
     * @param vlen
     *          length of value.
     * @throws IOException
     *           Upon IO errors.
     *           <p>
     *           If an exception is thrown, the TFile will be in an inconsistent
     *           state. The only legitimate call after that would be close
     */
    public void append(byte[] key, int koff, int klen, byte[] value, int voff,
        int vlen) throws IOException {
      if ((koff | klen | (koff + klen) | (key.length - (koff + klen))) < 0) {
        throw new IndexOutOfBoundsException(
            "Bad key buffer offset-length combination.");
      }

      if ((voff | vlen | (voff + vlen) | (value.length - (voff + vlen))) < 0) {
        throw new IndexOutOfBoundsException(
            "Bad value buffer offset-length combination.");
      }

      try {
        DataOutputStream dosKey = prepareAppendKey(klen);
        try {
          ++errorCount;
          dosKey.write(key, koff, klen);
          --errorCount;
        } finally {
          dosKey.close();
        }

View on GitHub (pinned to 2add963021)

Solutions

  1. Validate koff >= 0, klen >= 0, and koff + klen <= key.length before calling append
  2. Prefer the simple overload append(byte[] key, byte[] value) when the whole arrays are the record
  3. Where lengths come from external input, reject or clamp them at the parse boundary with an explicit error message

Example fix

// before
writer.append(keyBuf, keyOff, keyLen, valBuf, valOff, valLen);

// after
if (keyOff < 0 || keyLen < 0 || keyOff + keyLen > keyBuf.length
    || valOff < 0 || valLen < 0 || valOff + valLen > valBuf.length) {
  throw new IllegalArgumentException("Bad offset/length");
}
writer.append(keyBuf, keyOff, keyLen, valBuf, valOff, valLen);
Defensive patterns

Strategy: validation

Validate before calling

static void checkKeyBounds(byte[] key, int koff, int klen) {
  if (key == null || koff < 0 || klen < 0 || koff + klen < 0 || koff + klen > key.length) {
    throw new IllegalArgumentException("bad key bounds: off=" + koff + " len=" + klen + " buf=" + (key == null ? -1 : key.length));
  }
}
checkKeyBounds(key, koff, klen);
writer.append(key, koff, klen, value, voff, vlen);

Try / catch

catch (IndexOutOfBoundsException e) {
  // nothing was written; fix or drop the record and continue with the next one
}

Prevention

When it happens

Trigger: Passing lengths parsed from external data (e.g. unsigned fields read as negative ints), an offset computed as key.length - klen with wrong signs, or klen larger than the remaining buffer from koff.

Common situations: Deserializing lengths from network/file input without range checks; copying code from ByteBuffer-based loops into array-based append calls; off-by-one errors where klen includes a trailing byte; arithmetic that silently overflows int for large buffers.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e1a1198646885eaf. Report an issue: GitHub.