apache/hadoop · error · RuntimeException

Invalid configuration: maxSingleShuffleLimit should be less

Error message

Invalid configuration: maxSingleShuffleLimit should be less than mergeThreshold maxSingleShuffleLimit: {maxSingleShuffleLimit}mergeThreshold: {mergeThreshold}

What it means

MergeManagerImpl derives mergeThreshold = memoryLimit * mapreduce.reduce.shuffle.merge.percent (default 0.66f) — the used-memory level that triggers a merge to free memory — and maxSingleShuffleLimit = memoryLimit * mapreduce.reduce.shuffle.memory.limit.percent (default 0.25f). If one shuffle could fill memory past the merge trigger, memory could never be reclaimed, so the constructor throws RuntimeException and the reduce task fails at startup.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java:215

      maxSingleShuffleLimitConfiged = Integer.MAX_VALUE;
      LOG.info("The max number of bytes for a single in-memory shuffle cannot" +
          " be larger than Integer.MAX_VALUE. Setting it to Integer.MAX_VALUE");
    }
    this.maxSingleShuffleLimit = maxSingleShuffleLimitConfiged;
    this.memToMemMergeOutputsThreshold =
        jobConf.getInt(MRJobConfig.REDUCE_MEMTOMEM_THRESHOLD, ioSortFactor);
    this.mergeThreshold = (long)(this.memoryLimit * 
                          jobConf.getFloat(
                            MRJobConfig.SHUFFLE_MERGE_PERCENT,
                            MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT));
    LOG.info("MergerManager: memoryLimit=" + memoryLimit + ", " +
             "maxSingleShuffleLimit=" + maxSingleShuffleLimit + ", " +
             "mergeThreshold=" + mergeThreshold + ", " + 
             "ioSortFactor=" + ioSortFactor + ", " +
             "memToMemMergeOutputsThreshold=" + memToMemMergeOutputsThreshold);

    if (this.maxSingleShuffleLimit >= this.mergeThreshold) {
      throw new RuntimeException("Invalid configuration: "
          + "maxSingleShuffleLimit should be less than mergeThreshold "
          + "maxSingleShuffleLimit: " + this.maxSingleShuffleLimit
          + "mergeThreshold: " + this.mergeThreshold);
    }

    boolean allowMemToMemMerge = 
      jobConf.getBoolean(MRJobConfig.REDUCE_MEMTOMEM_ENABLED, false);
    if (allowMemToMemMerge) {
      this.memToMemMerger = 
        new IntermediateMemoryToMemoryMerger(this,
                                             memToMemMergeOutputsThreshold);
      this.memToMemMerger.start();
    } else {
      this.memToMemMerger = null;
    }
    
    this.inMemoryMerger = createInMemoryMerger();
    this.inMemoryMerger.start();

View on GitHub (pinned to 2add963021)

Solutions

1. Lower mapreduce.reduce.shuffle.memory.limit.percent below mapreduce.reduce.shuffle.merge.percent (defaults 0.25 < 0.66 always satisfy this).
2. Or raise mapreduce.reduce.shuffle.merge.percent toward (but above) the limit percent so merging triggers later.
3. If large outputs must stay in memory, increase reducer heap rather than the limit fraction.

Example fix

// before: single-shuffle limit above the merge threshold -> RuntimeException
conf.setFloat("mapreduce.reduce.shuffle.memory.limit.percent", 0.8f); // merge.percent = 0.66
// after: keep the limit strictly below the merge threshold
conf.setFloat("mapreduce.reduce.shuffle.memory.limit.percent", 0.5f); // < 0.66

// Validate the fraction pair before job submission
float limit = conf.getFloat("mapreduce.reduce.shuffle.memory.limit.percent", 0.25f);
float merge = conf.getFloat("mapreduce.reduce.shuffle.merge.percent", 0.66f);
if (limit >= merge) { throw new IllegalArgumentException("memory.limit.percent (" + limit + ") must stay below merge.percent (" + merge + ")"); }

When it happens

Trigger: mapreduce.reduce.shuffle.memory.limit.percent >= mapreduce.reduce.shuffle.merge.percent, e.g. 0.8 with the default merge percent of 0.66; also reachable by raising merge percent below an existing limit percent.

Common situations: Aggressive tuning to keep large map outputs in memory without lowering the merge threshold correspondingly; config pairs copied from different tuning guides.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc
• {maxRedPer}: mapreduce.reduce.input.buffer.percent must be a
• Invalid timeout [timeout = {connectionTimeout} ms]
• Rec# {recNo}: Failed to skip past key of length: {currentKey

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e7021b5f57671cbc. Report an issue: GitHub.