apache/hadoop · error · RuntimeException

Internal error

Error message

Internal error

What it means

RuntimeException("Internal error") from TFileIndex(DataInput): the first-key entry size decoded to <= 0 (meaning 'no first key') but entryCount != 0. The format invariant is that an index with at least one block must carry a first key; an empty first key is only legal for a zero-block (empty) file. Violating that pairing means the index header itself is inconsistent.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2218

        for (int i = 0; i < entryCount; i++) {
          size = Utils.readVInt(in);
          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {
            throw new IOException("Index entry size out of range: " + size);
          }
          if (buffer.length < size) {
            buffer = new byte[size];
          }
          in.readFully(buffer, 0, size);
          TFileIndexEntry idx =
              new TFileIndexEntry(new DataInputStream(new ByteArrayInputStream(
                  buffer, 0, size)));
          index.add(idx);
          sum += idx.entries();
          recordNumIndex.add(sum);
        }
      } else {
        if (entryCount != 0) {
          throw new RuntimeException("Internal error");
        }
      }
      this.comparator = comparator;
    }

    /**
     * @param key
     *          input key.
     * @return the ID of the first block that contains key >= input key. Or -1
     *         if no such block exists.
     */
    public int lowerBound(RawComparable key) {
      if (comparator == null) {
        throw new RuntimeException("Cannot search in unsorted TFile");
      }

      if (firstKey == null) {
        return -1; // not found

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as unreadable corruption: restore from replica/source; there is no caller-side workaround.
2. If the file came from a third-party writer, regenerate it with the Apache Hadoop TFile.Writer.
3. Report upstream only if reproducible with files written by this codebase's own Writer (that would be a genuine bug).

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (RuntimeException e) {
  if ("Internal error".equals(e.getMessage())) { /* index invariant broken: file unusable */ }
  else throw e;
}

When it happens

Trigger: A corrupt or malformed index where size <= 0 coexists with a nonzero block count: zeroed-out index bytes after truncation, or a hand-crafted/nonconforming TFile. It is explicitly labeled 'Internal error' because well-formed writers cannot produce this combination.

Common situations: Files damaged in ways that zero header bytes (some filesystem corruption modes), or reading experimental output from third-party TFile writers that don't emit the first key correctly.

Related errors

• Cannot find matching key in block.
• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}
• Index entry size out of range: {size}
• Index entry key length out of range: {len}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/89fd34c4a133bc90. Report an issue: GitHub.