apache/hadoop · error · IllegalArgumentException

Invalid value for mapreduce.reduce.shuffle.input.buffer.perc

Error message

Invalid value for mapreduce.reduce.shuffle.input.buffer.percent: {maxInMemCopyUse}

What it means

MergeManagerImpl's constructor reads mapreduce.reduce.shuffle.input.buffer.percent (fraction of the reducer's memory budget usable for in-memory shuffle data, default 0.70) and requires it within [0.0, 1.0]; otherwise it throws IllegalArgumentException and the reduce task fails at startup. memoryLimit is later computed as maxMemory (or mapreduce.reduce.memory.total.bytes) times this fraction.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java:169

    
    this.reporter = reporter;
    this.codec = codec;
    this.combinerClass = combinerClass;
    this.combineCollector = combineCollector;
    this.reduceCombineInputCounter = reduceCombineInputCounter;
    this.spilledRecordsCounter = spilledRecordsCounter;
    this.mergedMapOutputsCounter = mergedMapOutputsCounter;
    this.mapOutputFile = mapOutputFile;
    this.mapOutputFile.setConf(jobConf);
    
    this.localFS = localFS;
    this.rfs = ((LocalFileSystem)localFS).getRaw();
    
    final float maxInMemCopyUse =
      jobConf.getFloat(MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT,
          MRJobConfig.DEFAULT_SHUFFLE_INPUT_BUFFER_PERCENT);
    if (maxInMemCopyUse > 1.0 || maxInMemCopyUse < 0.0) {
      throw new IllegalArgumentException("Invalid value for " +
          MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT + ": " +
          maxInMemCopyUse);
    }

    // Allow unit tests to fix Runtime memory
    this.memoryLimit = (long)(jobConf.getLong(
        MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES,
        Runtime.getRuntime().maxMemory()) * maxInMemCopyUse);

    this.ioSortFactor = jobConf.getInt(MRJobConfig.IO_SORT_FACTOR,
        MRJobConfig.DEFAULT_IO_SORT_FACTOR);

    final float singleShuffleMemoryLimitPercent =
        jobConf.getFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT,
            DEFAULT_SHUFFLE_MEMORY_LIMIT_PERCENT);
    if (singleShuffleMemoryLimitPercent < 0.0f
        || singleShuffleMemoryLimitPercent > 1.0f) {
      throw new IllegalArgumentException("Invalid value for "

View on GitHub (pinned to 2add963021)

Solutions

1. Set mapreduce.reduce.shuffle.input.buffer.percent to a plain float between 0.0 and 1.0 (default 0.70).
2. Remove the override to fall back to the default.
3. If more shuffle memory is the goal, raise the reducer heap / mapreduce.reduce.memory.total.bytes instead of pushing the fraction above 1.

Example fix

<!-- before: percent instead of fraction -->
<property><name>mapreduce.reduce.shuffle.input.buffer.percent</name><value>70</value></property>
<!-- after: fraction of heap -->
<property><name>mapreduce.reduce.shuffle.input.buffer.percent</name><value>0.70</value></property>

// Validate before job submission
float p = conf.getFloat("mapreduce.reduce.shuffle.input.buffer.percent", 0.70f);
if (p < 0.0f || p > 1.0f) { throw new IllegalArgumentException("mapreduce.reduce.shuffle.input.buffer.percent must be a fraction in [0,1]: " + p); }

When it happens

Trigger: The job conf contains a value like '1.5', '-0.1', '70' (percent instead of fraction), or '0.7f' for mapreduce.reduce.shuffle.input.buffer.percent.

Common situations: Operators pasting a percentage where a fraction is expected; templated config files substituting wrong values; copies from documentation of other shuffle settings that use bytes or percents.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc
• Invalid configuration: maxSingleShuffleLimit should be less
• {maxRedPer}: mapreduce.reduce.input.buffer.percent must be a
• Invalid timeout [timeout = {connectionTimeout} ms]
• Negative key-length not allowed: {keyLength} for {key}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5170ace946ea9505. Report an issue: GitHub.