apache/hadoop · error · IOException

Key length out of range: {klen}

Error message

Key length out of range: {klen}

What it means

Thrown by TFile.Reader.Scanner.Entry.checkKey() while decoding the current record from a data block. Every TFile record starts with a VInt-encoded key length; the reader validates it against MAX_KEY_SIZE (64KB, TFile.java:154) and throws IOException when klen < 0 or klen > 65536. In practice it means the bytes at the reader's current block position are not a well-formed TFile record stream.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:1617

        return (currentLocation.compareTo(endLocation) >= 0);
      }

      /**
       * check whether we have already successfully obtained the key. It also
       * initializes the valueInputStream.
       */
      void checkKey() throws IOException {
        if (klen >= 0) return;
        if (atEnd()) {
          throw new EOFException("No key-value to read");
        }
        klen = -1;
        vlen = -1;
        valueChecked = false;

        klen = Utils.readVInt(blkReader);
        if (klen < 0 || klen > MAX_KEY_SIZE) {
          throw new IOException("Key length out of range: " + klen);
        }
        blkReader.readFully(keyBuffer, 0, klen);
        valueBufferInputStream.reset(blkReader);
        if (valueBufferInputStream.isLastChunk()) {
          vlen = valueBufferInputStream.getRemain();
        }
      }

      /**
       * Get an entry to access the key and value.
       * 
       * @return The Entry object to access the key and value.
       * @throws IOException raised on errors performing I/O.
       */
      public Entry entry() throws IOException {
        checkKey();
        return new Entry();
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify file integrity: run 'hdfs fsck -blocks' on the path and re-fetch a healthy replica, or compare against the original copy.
2. Regenerate the TFile from source data, ensuring Writer.close() (and the underlying stream close) completes before any reader opens it.
3. Confirm the file is really a TFile: BCFile magic check happens earlier, so a version/magic failure points elsewhere; a klen failure here points to data-block corruption specifically.
4. Write future files with BCFile checksums enabled so corruption is detected at the chunk level instead of surfacing as bogus lengths.

Example fix

// before
while (!scanner.atEnd()) { scanner.advance(); consume(scanner.entry()); }
// after
try {
  while (!scanner.atEnd()) { scanner.advance(); consume(scanner.entry()); }
} catch (IOException e) {
  // record stream is corrupt: stop, quarantine the file, rebuild from source
  quarantine(path); throw e;
}

try {
  while (!scanner.atEnd()) { scanner.advance(); consume(scanner.entry()); }
} catch (IOException e) {
  // bogus klen = data-block corruption: quarantine, do not retry same bytes
  quarantine(path); throw e;
}

When it happens

Trigger: Scanner iteration (atEnd(), advance(), any getKey*/getValue* call that forces checkKey()) where Utils.readVInt(blkReader) at the current block offset decodes to a negative value or a value above 65536. Typical causes: truncated or bit-rotted data block, a data block desynchronized from record boundaries, or opening a file that is not a TFile at all.

Common situations: Bad HDFS disks / corrupted replicas, jobs killed before TFile.Writer.close() so the file was never finalized, copying a SequenceFile or text file into a path then opening it as TFile, or reading through an unchecksummed transfer. Also seen when hand-rolling code that seeks into the middle of a TFile.

Related errors

• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}
• Index entry size out of range: {size}
• Index entry key length out of range: {len}
• Cannot find matching key in block.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/16939c0f0d403f1c. Report an issue: GitHub.