apache/hadoop · error · IOException

Index entry size out of range: {size}

Error message

Index entry size out of range: {size}

What it means

IOException from the per-block loop of TFileIndex(DataInput): for one of the entryCount index entries, the VInt size decodes to a negative value or exceeds MAX_INDEX_ENTRY_SIZE (64KB + 16). Each entry describes one data block (its first key plus entry count), so an out-of-range size means the block index region is corrupt from that entry onward.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2203

          throw new IOException("First key entry size out of range: " + size);
        }
        byte[] buffer = new byte[size];
        in.readFully(buffer);
        DataInputStream firstKeyInputStream =
            new DataInputStream(new ByteArrayInputStream(buffer, 0, size));

        int firstKeyLength = Utils.readVInt(firstKeyInputStream);
        if (firstKeyLength < 0 || firstKeyLength > MAX_KEY_SIZE) {
          throw new IOException("First key length out of range: "
              + firstKeyLength);
        }
        firstKey = new ByteArray(new byte[firstKeyLength]);
        firstKeyInputStream.readFully(firstKey.buffer());

        for (int i = 0; i < entryCount; i++) {
          size = Utils.readVInt(in);
          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {
            throw new IOException("Index entry size out of range: " + size);
          }
          if (buffer.length < size) {
            buffer = new byte[size];
          }
          in.readFully(buffer, 0, size);
          TFileIndexEntry idx =
              new TFileIndexEntry(new DataInputStream(new ByteArrayInputStream(
                  buffer, 0, size)));
          index.add(idx);
          sum += idx.entries();
          recordNumIndex.add(sum);
        }
      } else {
        if (entryCount != 0) {
          throw new RuntimeException("Internal error");
        }
      }
      this.comparator = comparator;

View on GitHub (pinned to 2add963021)

Solutions

1. Restore or regenerate the file; the index is not repairable in place.
2. Verify transfers complete (length + checksum) before publishing a TFile.
3. Keep writes atomic (temp file + rename) so partial indexes are never visible.

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (IOException e) {
  // block-index entry undecodable: restore/regenerate
}

When it happens

Trigger: Reading a TFile whose block-index array is damaged past the first entry: partial index writes, mid-file corruption, or a truncated file where the loop tries to read more entries than the bytes present (subsequent readFully would fail, but a bogus size fails here first). Note entryCount is only a loop bound; garbage sizes are caught per-entry.

Common situations: Large multi-block files where corruption lands mid-index; killed jobs; files copied with interrupted transfers that nonetheless preserved length fields.

Related errors

• First key entry size out of range: {size}
• Key length out of range: {klen}
• First key length out of range: {firstKeyLength}
• Index entry key length out of range: {len}
• Cannot find matching key in block.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7ccbf7e4f1351eb6. Report an issue: GitHub.