apache/hadoop · error · IOException

Invalid header on split file

Error message

Invalid header on split file

What it means

readSplitMetaInfo requires the first bytes of job.splitmetainfo to equal JobSplit.META_SPLIT_FILE_HEADER; a mismatch throws before anything else is read. It means the file is truncated, corrupted, or not a split metadata file at all.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java:60

public class SplitMetaInfoReader {
  
  public static JobSplit.TaskSplitMetaInfo[] readSplitMetaInfo(
      JobID jobId, FileSystem fs, Configuration conf, Path jobSubmitDir) 
  throws IOException {
    long maxMetaInfoSize = conf.getLong(MRJobConfig.SPLIT_METAINFO_MAXSIZE,
        MRJobConfig.DEFAULT_SPLIT_METAINFO_MAXSIZE);
    Path metaSplitFile = JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir);
    String jobSplitFile = JobSubmissionFiles.getJobSplitFile(jobSubmitDir).toString();
    FileStatus fStatus = fs.getFileStatus(metaSplitFile);
    if (maxMetaInfoSize > 0 && fStatus.getLen() > maxMetaInfoSize) {
      throw new IOException("Split metadata size exceeded " +
          maxMetaInfoSize +". Aborting job " + jobId);
    }
    FSDataInputStream in = fs.open(metaSplitFile);
    byte[] header = new byte[JobSplit.META_SPLIT_FILE_HEADER.length];
    in.readFully(header);
    if (!Arrays.equals(JobSplit.META_SPLIT_FILE_HEADER, header)) {
      throw new IOException("Invalid header on split file");
    }
    int vers = WritableUtils.readVInt(in);
    if (vers != JobSplit.META_SPLIT_VERSION) {
      in.close();
      throw new IOException("Unsupported split version " + vers);
    }
    int numSplits = WritableUtils.readVInt(in); //TODO: check for insane values
    JobSplit.TaskSplitMetaInfo[] allSplitMetaInfo = 
      new JobSplit.TaskSplitMetaInfo[numSplits];
    for (int i = 0; i < numSplits; i++) {
      JobSplit.SplitMetaInfo splitMetaInfo = new JobSplit.SplitMetaInfo();
      splitMetaInfo.readFields(in);
      JobSplit.TaskSplitIndex splitIndex = new JobSplit.TaskSplitIndex(
          jobSplitFile, 
          splitMetaInfo.getStartOffset());
      allSplitMetaInfo[i] = new JobSplit.TaskSplitMetaInfo(splitIndex, 
          splitMetaInfo.getLocations(), 
          splitMetaInfo.getInputDataLength());

View on GitHub (pinned to 2add963021)

Solutions

1. Resubmit the job into a fresh staging directory and let submission complete cleanly
2. Give each job a unique staging dir (mapreduce.job.staging-dir per user is the default; avoid sharing across concurrent jobs)
3. If corruption recurs, check NameNode and local disk health for the staging area

FileStatus st = fs.getFileStatus(JobSubmissionFiles.getJobSplitMetaFile(submitDir));
if (st.getLen() < JobSplit.META_SPLIT_FILE_HEADER.length) {
  throw new IOException("Truncated split metainfo at " + submitDir);
}

try {
  job.submit();
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Invalid header")) {
    cleanStagingDir(stagingDir); // wipe and resubmit once
    job.submit();
  } else { throw e; }
}

When it happens

Trigger: A half-written job.splitmetainfo from an interrupted job submission; two jobs sharing one staging directory and overwriting each other's files; disk or NameNode corruption; jobSubmitDir pointing at a directory that never held a real submission.

Common situations: Resubmitting into a reused staging dir after a crashed client; scripts that rsync or copy partial submit dirs; NN/disk faults on the staging area; hand-crafted job submit dirs used with the AM.

Related errors

• Split metadata size exceeded {maxMetaInfoSize}. Aborting job
• Unsupported split version {vers}
• Invalid specification for distributed-cache artifacts of typ
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• No application program defined.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/87fc5306691d307e. Report an issue: GitHub.