apache/hadoop · error · IOException

Index entry key length out of range: {len}

Error message

Index entry key length out of range: {len}

What it means

IOException from TFileIndexEntry(DataInput), which decodes one block-index entry: the VInt key length must satisfy 0 <= len <= MAX_KEY_SIZE (64KB) before the key bytes are read. An out-of-range length means this index entry's bytes are corrupt — allocation is refused before readFully, so bogus lengths fail here instead of exhausting memory.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2348

        Utils.writeVInt(out, dob.getLength());
        out.write(dob.getData(), 0, dob.getLength());
      }
    }
  }

  /**
   * TFile Data Index entry. We should try to make the memory footprint of each
   * index entry as small as possible.
   */
  static final class TFileIndexEntry implements RawComparable {
    final byte[] key;
    // count of <key, value> entries in the block.
    final long kvEntries;

    public TFileIndexEntry(DataInput in) throws IOException {
      int len = Utils.readVInt(in);
      if (len < 0 || len > MAX_KEY_SIZE) {
        throw new IOException("Index entry key length out of range: " + len);
      }
      key = new byte[len];
      in.readFully(key, 0, len);
      kvEntries = Utils.readVLong(in);
    }

    // default entry, without any padding
    public TFileIndexEntry(byte[] newkey, int offset, int len, long entries) {
      key = new byte[len];
      System.arraycopy(newkey, offset, key, 0, len);
      this.kvEntries = entries;
    }

    @Override
    public byte[] buffer() {
      return key;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Restore the file from a healthy replica or regenerate from source; index entries cannot be reconstituted.
2. fsck / verify checksums at the storage layer before re-reading.
3. Ensure complete file transfers and atomic publishes so index tails are never partial.

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (IOException e) {
  // index entry key length undecodable: restore/regenerate file
}

When it happens

Trigger: Parsing any of the per-block index entries during TFile.Reader construction where the length prefix decodes negative or above 65536: corrupted index region, wrong stream position, or a file whose index was truncated mid-entry.

Common situations: Block-level index corruption on large files, transfers that truncate the index tail, or nonconforming writer implementations.

Related errors

• First key length out of range: {firstKeyLength}
• Key length out of range: {klen}
• First key entry size out of range: {size}
• Index entry size out of range: {size}
• Cannot find matching key in block.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c3e8ce1fa7d629c8. Report an issue: GitHub.