apache/hadoop · error · RuntimeException

{maxRedPer}: mapreduce.reduce.input.buffer.percent must be a

Error message

{maxRedPer}: mapreduce.reduce.input.buffer.percent must be a float between 0 and 1.0

What it means

MergeManagerImpl.getMaxInMemReduceLimit() reads mapreduce.reduce.input.buffer.percent — the fraction of the shuffle memory limit used to retain map outputs in memory while reduce() runs (0 disables retention, default 0) — and requires a float within [0.0, 1.0], throwing RuntimeException when the reduce phase starts otherwise.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java:688

      final int vlen = vb.getLength() - vp;
      value.reset(vb.getData(), vp, vlen);
      bytesRead += vlen;
    }
    public long getPosition() throws IOException {
      return bytesRead;
    }

    public void close() throws IOException {
      kvIter.close();
    }
  }

  @VisibleForTesting
  final long getMaxInMemReduceLimit() {
    final float maxRedPer =
        jobConf.getFloat(MRJobConfig.REDUCE_INPUT_BUFFER_PERCENT, 0f);
    if (maxRedPer > 1.0 || maxRedPer < 0.0) {
      throw new RuntimeException(maxRedPer + ": "
          + MRJobConfig.REDUCE_INPUT_BUFFER_PERCENT
          + " must be a float between 0 and 1.0");
    }
    return (long)(memoryLimit * maxRedPer);
  }

  private RawKeyValueIterator finalMerge(JobConf job, FileSystem fs,
                                       List<InMemoryMapOutput<K,V>> inMemoryMapOutputs,
                                       List<CompressAwarePath> onDiskMapOutputs
                                       ) throws IOException {
    LOG.info("finalMerge called with " +
        inMemoryMapOutputs.size() + " in-memory map-outputs and " +
        onDiskMapOutputs.size() + " on-disk map-outputs");
    final long maxInMemReduce = getMaxInMemReduceLimit();
    // merge config params
    Class<K> keyClass = (Class<K>)job.getMapOutputKeyClass();
    Class<V> valueClass = (Class<V>)job.getMapOutputValueClass();
    boolean keepInputs = job.getKeepFailedTaskFiles();

View on GitHub (pinned to 2add963021)

Solutions

1. Set mapreduce.reduce.input.buffer.percent to a float between 0.0 and 1.0.
2. Use 0.0 to disable keeping map outputs in memory during reduce (safest when heap is tight).
3. Remove the override to use the default of 0.

Example fix

<!-- before -->
<property><name>mapreduce.reduce.input.buffer.percent</name><value>1.2</value></property>
<!-- after -->
<property><name>mapreduce.reduce.input.buffer.percent</name><value>0.0</value></property>

// Validate before job submission
float p = conf.getFloat("mapreduce.reduce.input.buffer.percent", 0f);
if (p < 0.0f || p > 1.0f) { throw new IllegalArgumentException("mapreduce.reduce.input.buffer.percent must be a fraction in [0,1]: " + p); }

When it happens

Trigger: Job conf contains a value such as '2', '1.5', or '-0.1' for mapreduce.reduce.input.buffer.percent.

Common situations: Tuning guides suggesting values above 1 for more aggressive in-memory reduce; copy-paste of a percent instead of a fraction; leaving an experiment value in a shared template.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc
• Invalid configuration: maxSingleShuffleLimit should be less
• Invalid timeout [timeout = {connectionTimeout} ms]
• Negative key-length not allowed: {keyLength} for {key}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/772bbba9bf3cb92e. Report an issue: GitHub.