{"record":{"id":"fbd440da09f1fa04","repo":"apache/hadoop","slug":"too-many-spill-files-got-created-control-it-with","errorCode":null,"errorMessage":"Too many spill files got created, control it with mapreduce.task.spill.files.count.limit, current value: {spillFilesCountLimit}, current spill count: {numSpills}","messagePattern":"Too many spill files got created, control it with mapreduce\\.task\\.spill\\.files\\.count\\.limit, current value: (.+?), current spill count: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":2053,"sourceCode":"              + dst + \": couldn't create parent directory\");\n        }\n      }\n      \n      if (!src.renameTo(dst)) {\n        throw new IOException(\"Unable to rename \" + src + \" to \" + dst);\n      }\n    }\n\n    /**\n     * Increments numSpills local counter by taking into consideration\n     * the max limit on spill files being generated by the job.\n     * If limit is reached, this function throws an IOException\n     */\n    private void incrementNumSpills() throws IOException {\n      ++numSpills;\n      if(spillFilesCountLimit != SPILL_FILES_COUNT_UNBOUNDED_LIMIT_VALUE\n          && numSpills > spillFilesCountLimit) {\n        throw new IOException(\"Too many spill files got created, control it with \" +\n            \"mapreduce.task.spill.files.count.limit, current value: \" + spillFilesCountLimit +\n            \", current spill count: \" + numSpills);\n      }\n    }\n  } // MapOutputBuffer\n  \n  /**\n   * Exception indicating that the allocated sort buffer is insufficient\n   * to hold the current record.\n   */\n  @SuppressWarnings(\"serial\")\n  private static class MapBufferTooSmallException extends IOException {\n    public MapBufferTooSmallException(String s) {\n      super(s);\n    }\n  }\n\n  private <INKEY,INVALUE,OUTKEY,OUTVALUE>","sourceCodeStart":2035,"sourceCodeEnd":2071,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L2035-L2071","documentation":"A fail-fast guard against runaway map tasks: after each spill, incrementNumSpills() compares the spill count against mapreduce.task.spill.files.count.limit (default -1 = unbounded; when set it must be >= 1). Once numSpills exceeds the configured limit, the task fails with this IOException instead of flooding the node with spill files.","triggerScenarios":"mapreduce.task.spill.files.count.limit=N was set on the job or cluster and the mapper spilled more than N times: small io sort buffer (mapreduce.task.io.sort.mb), low spill threshold (mapreduce.task.io.sort.spill.percent), very large or very many map output records, or no combiner.","commonSituations":"Operators setting the limit to protect local disks; jobs whose map-side output volume grew after a data or upstream change; someone lowered io.sort.mb; bursty map tasks (map-only ETL) with the default 100MB buffer.","solutions":["Increase mapreduce.task.io.sort.mb (e.g. 256-512) and/or raise mapreduce.task.io.sort.spill.percent toward 0.9 so each spill holds more data","Set a combiner (or fix an ineffective one) so less data stays in the buffer","Raise mapreduce.task.spill.files.count.limit if the volume is legitimate and the node can absorb it","Check whether the map output volume is expected - data skew or a broken upstream producer may be the real cause"],"exampleFix":"# before\nmapreduce.task.io.sort.mb=100\nmapreduce.task.spill.files.count.limit=10\n\n# after\nmapreduce.task.io.sort.mb=512\nmapreduce.task.io.sort.spill.percent=0.9\nmapreduce.task.spill.files.count.limit=100\n# plus: job.setCombinerClass(IntSumReducer.class)","handlingStrategy":"validation","validationCode":"// before submission: sanity-check expected spill pressure against the configured limit\nlong sortBytes = conf.getInt(\"mapreduce.task.io.sort.mb\", 100) * 1024L * 1024L;\nint limit = conf.getInt(\"mapreduce.task.spill.files.count.limit\", -1);\nif (limit > 0 && expectedMapOutputBytesPerTask > limit * sortBytes * 0.8) {\n  throw new IllegalStateException(\"spill limit too low for expected map output; raise io.sort.mb or the limit\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size io.sort.mb from measured map output volumes (target O(10) spills, not hundreds)","Prefer combiners to reducing spill counts rather than raising limits","Benchmark new data shapes with a small job before full runs when a spill limit is set"],"tags":["hadoop","mapreduce","spill","io-sort-buffer","configuration","limits"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}