apache/hadoop · error · IOException

Too many spill files got created, control it with mapreduce.

Error message

Too many spill files got created, control it with mapreduce.task.spill.files.count.limit, current value: {spillFilesCountLimit}, current spill count: {numSpills}

What it means

A fail-fast guard against runaway map tasks: after each spill, incrementNumSpills() compares the spill count against mapreduce.task.spill.files.count.limit (default -1 = unbounded; when set it must be >= 1). Once numSpills exceeds the configured limit, the task fails with this IOException instead of flooding the node with spill files.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:2053

              + dst + ": couldn't create parent directory");
        }
      }
      
      if (!src.renameTo(dst)) {
        throw new IOException("Unable to rename " + src + " to " + dst);
      }
    }

    /**
     * Increments numSpills local counter by taking into consideration
     * the max limit on spill files being generated by the job.
     * If limit is reached, this function throws an IOException
     */
    private void incrementNumSpills() throws IOException {
      ++numSpills;
      if(spillFilesCountLimit != SPILL_FILES_COUNT_UNBOUNDED_LIMIT_VALUE
          && numSpills > spillFilesCountLimit) {
        throw new IOException("Too many spill files got created, control it with " +
            "mapreduce.task.spill.files.count.limit, current value: " + spillFilesCountLimit +
            ", current spill count: " + numSpills);
      }
    }
  } // MapOutputBuffer
  
  /**
   * Exception indicating that the allocated sort buffer is insufficient
   * to hold the current record.
   */
  @SuppressWarnings("serial")
  private static class MapBufferTooSmallException extends IOException {
    public MapBufferTooSmallException(String s) {
      super(s);
    }
  }

  private <INKEY,INVALUE,OUTKEY,OUTVALUE>

View on GitHub (pinned to 2add963021)

Solutions

Increase mapreduce.task.io.sort.mb (e.g. 256-512) and/or raise mapreduce.task.io.sort.spill.percent toward 0.9 so each spill holds more data
Set a combiner (or fix an ineffective one) so less data stays in the buffer
Raise mapreduce.task.spill.files.count.limit if the volume is legitimate and the node can absorb it
Check whether the map output volume is expected - data skew or a broken upstream producer may be the real cause

Example fix

# before
mapreduce.task.io.sort.mb=100
mapreduce.task.spill.files.count.limit=10

# after
mapreduce.task.io.sort.mb=512
mapreduce.task.io.sort.spill.percent=0.9
mapreduce.task.spill.files.count.limit=100
# plus: job.setCombinerClass(IntSumReducer.class)

Defensive patterns

Strategy: validation

Validate before calling

// before submission: sanity-check expected spill pressure against the configured limit
long sortBytes = conf.getInt("mapreduce.task.io.sort.mb", 100) * 1024L * 1024L;
int limit = conf.getInt("mapreduce.task.spill.files.count.limit", -1);
if (limit > 0 && expectedMapOutputBytesPerTask > limit * sortBytes * 0.8) {
  throw new IllegalStateException("spill limit too low for expected map output; raise io.sort.mb or the limit");
}

Prevention

Size io.sort.mb from measured map output volumes (target O(10) spills, not hundreds)
Prefer combiners to reducing spill counts rather than raising limits
Benchmark new data shapes with a small job before full runs when a spill limit is set

When it happens

Trigger: mapreduce.task.spill.files.count.limit=N was set on the job or cluster and the mapper spilled more than N times: small io sort buffer (mapreduce.task.io.sort.mb), low spill threshold (mapreduce.task.io.sort.spill.percent), very large or very many map output records, or no combiner.

Common situations: Operators setting the limit to protect local disks; jobs whose map-side output volume grew after a data or upstream change; someone lowered io.sort.mb; bursty map tasks (map-only ETL) with the default 100MB buffer.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fbd440da09f1fa04. Report an issue: GitHub.