apache/hadoop · error · IOException

Unsupported split version {vers}

Error message

Unsupported split version {vers}

What it means

After the header, readSplitMetaInfo reads a version integer that must equal JobSplit.META_SPLIT_VERSION; any other value throws with the offending number. The split metadata file was written by a Hadoop release whose split format differs from the one now reading it.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java:65

    long maxMetaInfoSize = conf.getLong(MRJobConfig.SPLIT_METAINFO_MAXSIZE,
        MRJobConfig.DEFAULT_SPLIT_METAINFO_MAXSIZE);
    Path metaSplitFile = JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir);
    String jobSplitFile = JobSubmissionFiles.getJobSplitFile(jobSubmitDir).toString();
    FileStatus fStatus = fs.getFileStatus(metaSplitFile);
    if (maxMetaInfoSize > 0 && fStatus.getLen() > maxMetaInfoSize) {
      throw new IOException("Split metadata size exceeded " +
          maxMetaInfoSize +". Aborting job " + jobId);
    }
    FSDataInputStream in = fs.open(metaSplitFile);
    byte[] header = new byte[JobSplit.META_SPLIT_FILE_HEADER.length];
    in.readFully(header);
    if (!Arrays.equals(JobSplit.META_SPLIT_FILE_HEADER, header)) {
      throw new IOException("Invalid header on split file");
    }
    int vers = WritableUtils.readVInt(in);
    if (vers != JobSplit.META_SPLIT_VERSION) {
      in.close();
      throw new IOException("Unsupported split version " + vers);
    }
    int numSplits = WritableUtils.readVInt(in); //TODO: check for insane values
    JobSplit.TaskSplitMetaInfo[] allSplitMetaInfo = 
      new JobSplit.TaskSplitMetaInfo[numSplits];
    for (int i = 0; i < numSplits; i++) {
      JobSplit.SplitMetaInfo splitMetaInfo = new JobSplit.SplitMetaInfo();
      splitMetaInfo.readFields(in);
      JobSplit.TaskSplitIndex splitIndex = new JobSplit.TaskSplitIndex(
          jobSplitFile, 
          splitMetaInfo.getStartOffset());
      allSplitMetaInfo[i] = new JobSplit.TaskSplitMetaInfo(splitIndex, 
          splitMetaInfo.getLocations(), 
          splitMetaInfo.getInputDataLength());
    }
    in.close();
    return allSplitMetaInfo;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Submit from a client whose Hadoop version matches the cluster (check hadoop classpath on the gateway)
2. Clear stale staging directories after an upgrade so old metainfo files are not picked up
3. Complete rolling upgrades before resubmitting jobs that failed mid-upgrade

if (!VersionInfo.getVersion().equals(clusterVersion)) {
  log.warn("Client {} vs cluster {} — split metainfo version may mismatch",
      VersionInfo.getVersion(), clusterVersion);
}

try {
  job.submit();
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).startsWith("Unsupported split version")) {
    throw new IOException("Client/cluster Hadoop version mismatch writing split metainfo", e);
  }
  throw e;
}

When it happens

Trigger: A client on Hadoop version X writes job.splitmetainfo, the ResourceManager/AM on version Y reads it during a rolling upgrade or from a mis-versioned gateway; stale metainfo from an earlier cluster version reused via a shared staging dir.

Common situations: Gateway machines with newer/older hadoop-mapreduce-client jars than the cluster; rolling upgrades where old AMs read new clients' submissions; staging dirs reused across an upgrade.

Related errors

• Split metadata size exceeded {maxMetaInfoSize}. Aborting job
• Invalid header on split file
• Not yet implemented.
• Unrecognized priority: {}
• Unrecognized task type: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e82b0c2ab44771cf. Report an issue: GitHub.