apache/hadoop · error · IOException

Split metadata size exceeded {maxMetaInfoSize}. Aborting job

Error message

Split metadata size exceeded {maxMetaInfoSize}. Aborting job {jobId}

What it means

At job start, SplitMetaInfoReader.readSplitMetaInfo compares the size of job.splitmetainfo against mapreduce.job.split.metainfo.maxsize (default 10,000,000 bytes) and aborts the job when the file is larger. It is a guardrail against pathologically large split lists overwhelming the AM.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java:53

/**
 * A utility that reads the split meta info and creates
 * split meta info objects
 */
@InterfaceAudience.Private
@InterfaceStability.Unstable
public class SplitMetaInfoReader {
  
  public static JobSplit.TaskSplitMetaInfo[] readSplitMetaInfo(
      JobID jobId, FileSystem fs, Configuration conf, Path jobSubmitDir) 
  throws IOException {
    long maxMetaInfoSize = conf.getLong(MRJobConfig.SPLIT_METAINFO_MAXSIZE,
        MRJobConfig.DEFAULT_SPLIT_METAINFO_MAXSIZE);
    Path metaSplitFile = JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir);
    String jobSplitFile = JobSubmissionFiles.getJobSplitFile(jobSubmitDir).toString();
    FileStatus fStatus = fs.getFileStatus(metaSplitFile);
    if (maxMetaInfoSize > 0 && fStatus.getLen() > maxMetaInfoSize) {
      throw new IOException("Split metadata size exceeded " +
          maxMetaInfoSize +". Aborting job " + jobId);
    }
    FSDataInputStream in = fs.open(metaSplitFile);
    byte[] header = new byte[JobSplit.META_SPLIT_FILE_HEADER.length];
    in.readFully(header);
    if (!Arrays.equals(JobSplit.META_SPLIT_FILE_HEADER, header)) {
      throw new IOException("Invalid header on split file");
    }
    int vers = WritableUtils.readVInt(in);
    if (vers != JobSplit.META_SPLIT_VERSION) {
      in.close();
      throw new IOException("Unsupported split version " + vers);
    }
    int numSplits = WritableUtils.readVInt(in); //TODO: check for insane values
    JobSplit.TaskSplitMetaInfo[] allSplitMetaInfo = 
      new JobSplit.TaskSplitMetaInfo[numSplits];
    for (int i = 0; i < numSplits; i++) {
      JobSplit.SplitMetaInfo splitMetaInfo = new JobSplit.SplitMetaInfo();

View on GitHub (pinned to 2add963021)

Solutions

If the split count is legitimate, raise mapreduce.job.split.metainfo.maxsize accordingly
Increase mapreduce.input.fileinputformat.split.minsize (or the per-node/per-rack variants with CombineFileInputFormat) so each map handles more data
Consolidate small files before the job — HAR, SequenceFile, Avro, or Parquet compaction

Example fix

// before: forced tiny splits -> millions of splits
conf.setLong("mapreduce.input.fileinputformat.split.minsize", 1L);
// after: block-sized splits, or raise the cap deliberately
conf.setLong("mapreduce.input.fileinputformat.split.minsize", 128L * 1024 * 1024);
conf.setLong("mapreduce.job.split.metainfo.maxsize", 50_000_000L);

Defensive patterns

Strategy: validation

Validate before calling

Job job = Job.getInstance(conf);
long splits = new TextInputFormat().getSplits(job).size();
long maxBytes = conf.getLong("mapreduce.job.split.metainfo.maxsize", 10_000_000L);
if (splits * 100L > maxBytes) { // rough per-split metadata estimate
  throw new IOException("~" + splits + " splits will exceed split metainfo cap; raise "
      + "mapreduce.job.split.metainfo.maxsize or increase split size");
}

Prevention

Estimate split count pre-submit when input is many small files
Compact small files or use CombineFileInputFormat before MapReduce
Set mapreduce.job.split.metainfo.maxsize deliberately when huge split counts are expected

When it happens

Trigger: An input split count in the millions — masses of small files, or split minsizes set far below the HDFS block size — inflating the per-split metadata (locations + offsets) past the cap; a size override of 0 disables the check only when configured as such.

Common situations: Directories of millions of small files (logs, CSV shards) fed to TextInputFormat; mapreduce.input.fileinputformat.split.minsize forced to 1 in an attempt to maximize parallelism; migration jobs scanning a deep HDFS tree.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c4900d98b3c45f53. Report an issue: GitHub.