apache/hadoop · error · IOException

First key length out of range: {firstKeyLength}

Error message

First key length out of range: {firstKeyLength}

What it means

IOException from TFileIndex(DataInput) while parsing inside the first-key entry blob: after size bytes are read and wrapped in a ByteArrayInputStream, the VInt firstKeyLength decoded from that blob is negative or exceeds MAX_KEY_SIZE (64KB). This indirection (size -> blob -> key length) means the blob itself is corrupt even though its outer size passed the earlier check.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2194

      // entryCount is derived from the file; only use it as a capacity hint,
      // bounded, so a corrupt value cannot force a huge pre-allocation. The
      // loops below are limited by the actual bytes available in the stream.
      int capacityHint = Math.max(0, Math.min(entryCount, INDEX_CAPACITY_HINT_CAP));
      index = new ArrayList<>(capacityHint);
      recordNumIndex = new ArrayList<>(capacityHint);
      int size = Utils.readVInt(in); // size for the first key entry.
      if (size > 0) {
        if (size > MAX_INDEX_ENTRY_SIZE) {
          throw new IOException("First key entry size out of range: " + size);
        }
        byte[] buffer = new byte[size];
        in.readFully(buffer);
        DataInputStream firstKeyInputStream =
            new DataInputStream(new ByteArrayInputStream(buffer, 0, size));

        int firstKeyLength = Utils.readVInt(firstKeyInputStream);
        if (firstKeyLength < 0 || firstKeyLength > MAX_KEY_SIZE) {
          throw new IOException("First key length out of range: "
              + firstKeyLength);
        }
        firstKey = new ByteArray(new byte[firstKeyLength]);
        firstKeyInputStream.readFully(firstKey.buffer());

        for (int i = 0; i < entryCount; i++) {
          size = Utils.readVInt(in);
          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {
            throw new IOException("Index entry size out of range: " + size);
          }
          if (buffer.length < size) {
            buffer = new byte[size];
          }
          in.readFully(buffer, 0, size);
          TFileIndexEntry idx =
              new TFileIndexEntry(new DataInputStream(new ByteArrayInputStream(
                  buffer, 0, size)));
          index.add(idx);

View on GitHub (pinned to 2add963021)

Solutions

1. Restore the file from a healthy replica or regenerate it from source data.
2. Verify integrity at the storage layer (checksums/fsck) before blaming the library.
3. Ensure only genuine TFiles (same version) are opened by this reader.

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (IOException e) {
  // first-key entry corrupt: quarantine and rebuild file
}

When it happens

Trigger: A first-key entry whose outer size is within 64KB+16 but whose inner VInt decodes to a negative value or > 65536 — random bytes where the length prefix should be, from truncation, bit rot, or a misaligned index read.

Common situations: Same family as other index corruption: interrupted close, bad replicas, reading with a wrong fileLength. Can also appear when a file written by a non-standard TFile implementation stores a different first-key layout.

Related errors

• Index entry key length out of range: {len}
• Key length out of range: {klen}
• First key entry size out of range: {size}
• Index entry size out of range: {size}
• Cannot find matching key in block.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d3446f0bc2df0b6e. Report an issue: GitHub.