apache/hadoop · error · IllegalArgumentException

Invalid value for mapreduce.reduce.shuffle.memory.limit.perc

Error message

Invalid value for mapreduce.reduce.shuffle.memory.limit.percent: {singleShuffleMemoryLimitPercent}

What it means

MergeManagerImpl's constructor validates mapreduce.reduce.shuffle.memory.limit.percent (maximum fraction of the shuffle memory that a single map output may occupy, default 0.25f) within [0.0, 1.0], throwing IllegalArgumentException otherwise. Map outputs larger than the resulting maxSingleShuffleLimit are copied straight to disk instead of into memory.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java:187

      throw new IllegalArgumentException("Invalid value for " +
          MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT + ": " +
          maxInMemCopyUse);
    }

    // Allow unit tests to fix Runtime memory
    this.memoryLimit = (long)(jobConf.getLong(
        MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES,
        Runtime.getRuntime().maxMemory()) * maxInMemCopyUse);

    this.ioSortFactor = jobConf.getInt(MRJobConfig.IO_SORT_FACTOR,
        MRJobConfig.DEFAULT_IO_SORT_FACTOR);

    final float singleShuffleMemoryLimitPercent =
        jobConf.getFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT,
            DEFAULT_SHUFFLE_MEMORY_LIMIT_PERCENT);
    if (singleShuffleMemoryLimitPercent < 0.0f
        || singleShuffleMemoryLimitPercent > 1.0f) {
      throw new IllegalArgumentException("Invalid value for "
          + MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT + ": "
          + singleShuffleMemoryLimitPercent);
    }

    usedMemory = 0L;
    commitMemory = 0L;
    long maxSingleShuffleLimitConfiged =
        (long)(memoryLimit * singleShuffleMemoryLimitPercent);
    if(maxSingleShuffleLimitConfiged > Integer.MAX_VALUE) {
      maxSingleShuffleLimitConfiged = Integer.MAX_VALUE;
      LOG.info("The max number of bytes for a single in-memory shuffle cannot" +
          " be larger than Integer.MAX_VALUE. Setting it to Integer.MAX_VALUE");
    }
    this.maxSingleShuffleLimit = maxSingleShuffleLimitConfiged;
    this.memToMemMergeOutputsThreshold =
        jobConf.getInt(MRJobConfig.REDUCE_MEMTOMEM_THRESHOLD, ioSortFactor);
    this.mergeThreshold = (long)(this.memoryLimit * 
                          jobConf.getFloat(

View on GitHub (pinned to 2add963021)

Solutions

1. Set mapreduce.reduce.shuffle.memory.limit.percent to a fraction in [0.0, 1.0] (default 0.25).
2. Use 0.0, not a negative number, to force all shuffle outputs to disk.
3. Keep it below mapreduce.reduce.shuffle.merge.percent (default 0.66) or construction fails with the maxSingleShuffleLimit check.

Example fix

<!-- before -->
<property><name>mapreduce.reduce.shuffle.memory.limit.percent</name><value>-1</value></property>
<!-- after: 0 keeps every map output on disk -->
<property><name>mapreduce.reduce.shuffle.memory.limit.percent</name><value>0.0</value></property>

// Validate before job submission
float p = conf.getFloat("mapreduce.reduce.shuffle.memory.limit.percent", 0.25f);
if (p < 0.0f || p > 1.0f) { throw new IllegalArgumentException("mapreduce.reduce.shuffle.memory.limit.percent must be a fraction in [0,1]: " + p); }

When it happens

Trigger: The job conf sets mapreduce.reduce.shuffle.memory.limit.percent to a value below 0.0 or above 1.0 (e.g. '-1' or '25').

Common situations: Tuning attempts that enter a percent where a fraction is required; disabling in-memory shuffle with '-1' (the correct disable value is 0.0).

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid configuration: maxSingleShuffleLimit should be less
• {maxRedPer}: mapreduce.reduce.input.buffer.percent must be a
• Invalid timeout [timeout = {connectionTimeout} ms]
• Negative key-length not allowed: {keyLength} for {key}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fa46efac7b5b293a. Report an issue: GitHub.