apache/hadoop · error · RuntimeException

Incompatible TFile fileVersion.

Error message

Incompatible TFile fileVersion.

What it means

RuntimeException from the TFileMeta read constructor: the Version parsed from the file header fails version.compatibleWith(TFile.API_VERSION) (currently 1.0). TFile deliberately gates reads on version compatibility so newer or incompatible formats fail fast at open time instead of misparsing later.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2076

    final Version version;
    private long recordCount;
    private final String strComparator;
    private final BytesComparator comparator;

    // ctor for writes
    TFileMeta(String comparator, Configuration conf) {
      // set fileVersion to API version when we create it.
      version = TFile.API_VERSION;
      recordCount = 0;
      strComparator = (comparator == null) ? "" : comparator;
      this.comparator = makeComparator(strComparator, conf);
    }

    // ctor for reads
    TFileMeta(DataInput in, Configuration conf) throws IOException {
      version = new Version(in);
      if (!version.compatibleWith(TFile.API_VERSION)) {
        throw new RuntimeException("Incompatible TFile fileVersion.");
      }
      recordCount = Utils.readVLong(in);
      strComparator = Utils.readString(in, MAX_COMPARATOR_LENGTH);
      comparator = makeComparator(strComparator, conf);
    }

    static BytesComparator makeComparator(String comparator) {
      return makeComparator(comparator, new Configuration());
    }

    @SuppressWarnings("unchecked")
    static BytesComparator makeComparator(String comparator,
        Configuration conf) {
      if (comparator.length() == 0) {
        // unsorted keys
        return null;
      }
      if (comparator.equals(COMPARATOR_MEMCMP)) {

View on GitHub (pinned to 2add963021)

Solutions

1. Read and write with compatible Hadoop versions: pin the cluster to one version through the upgrade, or upgrade readers first.
2. Verify the stream position (seek(0)) and fileLength passed to the Reader constructor match the actual file.
3. If the file legitimately comes from a newer format, convert it on a node that can read it (e.g. dump with TFileDumper and rewrite) before handing it to older readers.

Example fix

// before
TFile.Reader r = new TFile.Reader(in, len, conf); // may throw on version skew
// after
try {
  TFile.Reader r = new TFile.Reader(in, len, conf);
} catch (RuntimeException e) {
  if (e.getMessage().contains("fileVersion")) { /* writer/reader skew: convert or upgrade */ }
  else throw e;
}

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (RuntimeException e) {
  if (String.valueOf(e.getMessage()).contains("fileVersion")) {
    // version skew: route to conversion or fail with upgrade guidance
  } else { throw e; }
}

When it happens

Trigger: new TFile.Reader(fsdis, fileLength, conf) on a file whose embedded major/minor version is not compatible with 1.0: files produced by a newer TFile implementation, experimental formats, or a stream position/length that is wrong so the version short-pair decodes garbage.

Common situations: Rolling upgrades where writer nodes run a newer Hadoop than readers, copying files written by downstream forks that bumped the version, or passing a wrong fileLength / already-consumed FSDataInputStream so the constructor reads the version from the wrong offset.

Related errors

• Key length out of range: {klen}
• Buffer not enough to store the key
• Value length unknown.
• Buffer too small to hold value
• Attempt to examine value multiple times.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2948e894ef90fddc. Report an issue: GitHub.