{"record":{"id":"c4900d98b3c45f53","repo":"apache/hadoop","slug":"split-metadata-size-exceeded-maxmetainfosize-ab","errorCode":null,"errorMessage":"Split metadata size exceeded {maxMetaInfoSize}. Aborting job {jobId}","messagePattern":"Split metadata size exceeded (.+?)\\. Aborting job (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java","lineNumber":53,"sourceCode":"\n/**\n * A utility that reads the split meta info and creates\n * split meta info objects\n */\n@InterfaceAudience.Private\n@InterfaceStability.Unstable\npublic class SplitMetaInfoReader {\n  \n  public static JobSplit.TaskSplitMetaInfo[] readSplitMetaInfo(\n      JobID jobId, FileSystem fs, Configuration conf, Path jobSubmitDir) \n  throws IOException {\n    long maxMetaInfoSize = conf.getLong(MRJobConfig.SPLIT_METAINFO_MAXSIZE,\n        MRJobConfig.DEFAULT_SPLIT_METAINFO_MAXSIZE);\n    Path metaSplitFile = JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir);\n    String jobSplitFile = JobSubmissionFiles.getJobSplitFile(jobSubmitDir).toString();\n    FileStatus fStatus = fs.getFileStatus(metaSplitFile);\n    if (maxMetaInfoSize > 0 && fStatus.getLen() > maxMetaInfoSize) {\n      throw new IOException(\"Split metadata size exceeded \" +\n          maxMetaInfoSize +\". Aborting job \" + jobId);\n    }\n    FSDataInputStream in = fs.open(metaSplitFile);\n    byte[] header = new byte[JobSplit.META_SPLIT_FILE_HEADER.length];\n    in.readFully(header);\n    if (!Arrays.equals(JobSplit.META_SPLIT_FILE_HEADER, header)) {\n      throw new IOException(\"Invalid header on split file\");\n    }\n    int vers = WritableUtils.readVInt(in);\n    if (vers != JobSplit.META_SPLIT_VERSION) {\n      in.close();\n      throw new IOException(\"Unsupported split version \" + vers);\n    }\n    int numSplits = WritableUtils.readVInt(in); //TODO: check for insane values\n    JobSplit.TaskSplitMetaInfo[] allSplitMetaInfo = \n      new JobSplit.TaskSplitMetaInfo[numSplits];\n    for (int i = 0; i < numSplits; i++) {\n      JobSplit.SplitMetaInfo splitMetaInfo = new JobSplit.SplitMetaInfo();","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java#L35-L71","documentation":"At job start, SplitMetaInfoReader.readSplitMetaInfo compares the size of job.splitmetainfo against mapreduce.job.split.metainfo.maxsize (default 10,000,000 bytes) and aborts the job when the file is larger. It is a guardrail against pathologically large split lists overwhelming the AM.","triggerScenarios":"An input split count in the millions — masses of small files, or split minsizes set far below the HDFS block size — inflating the per-split metadata (locations + offsets) past the cap; a size override of 0 disables the check only when configured as such.","commonSituations":"Directories of millions of small files (logs, CSV shards) fed to TextInputFormat; mapreduce.input.fileinputformat.split.minsize forced to 1 in an attempt to maximize parallelism; migration jobs scanning a deep HDFS tree.","solutions":["If the split count is legitimate, raise mapreduce.job.split.metainfo.maxsize accordingly","Increase mapreduce.input.fileinputformat.split.minsize (or the per-node/per-rack variants with CombineFileInputFormat) so each map handles more data","Consolidate small files before the job — HAR, SequenceFile, Avro, or Parquet compaction"],"exampleFix":"// before: forced tiny splits -> millions of splits\nconf.setLong(\"mapreduce.input.fileinputformat.split.minsize\", 1L);\n// after: block-sized splits, or raise the cap deliberately\nconf.setLong(\"mapreduce.input.fileinputformat.split.minsize\", 128L * 1024 * 1024);\nconf.setLong(\"mapreduce.job.split.metainfo.maxsize\", 50_000_000L);","handlingStrategy":"validation","validationCode":"Job job = Job.getInstance(conf);\nlong splits = new TextInputFormat().getSplits(job).size();\nlong maxBytes = conf.getLong(\"mapreduce.job.split.metainfo.maxsize\", 10_000_000L);\nif (splits * 100L > maxBytes) { // rough per-split metadata estimate\n  throw new IOException(\"~\" + splits + \" splits will exceed split metainfo cap; raise \"\n      + \"mapreduce.job.split.metainfo.maxsize or increase split size\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Estimate split count pre-submit when input is many small files","Compact small files or use CombineFileInputFormat before MapReduce","Set mapreduce.job.split.metainfo.maxsize deliberately when huge split counts are expected"],"tags":["mapreduce","job-submission","splits","small-files","configuration"],"backgroundTag":"size-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}