{"record":{"id":"fa46efac7b5b293a","repo":"apache/hadoop","slug":"invalid-value-for-mapreduce-reduce-shuffle-memory","errorCode":null,"errorMessage":"Invalid value for mapreduce.reduce.shuffle.memory.limit.percent: {singleShuffleMemoryLimitPercent}","messagePattern":"Invalid value for mapreduce\\.reduce\\.shuffle\\.memory\\.limit\\.percent: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java","lineNumber":187,"sourceCode":"      throw new IllegalArgumentException(\"Invalid value for \" +\n          MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT + \": \" +\n          maxInMemCopyUse);\n    }\n\n    // Allow unit tests to fix Runtime memory\n    this.memoryLimit = (long)(jobConf.getLong(\n        MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES,\n        Runtime.getRuntime().maxMemory()) * maxInMemCopyUse);\n\n    this.ioSortFactor = jobConf.getInt(MRJobConfig.IO_SORT_FACTOR,\n        MRJobConfig.DEFAULT_IO_SORT_FACTOR);\n\n    final float singleShuffleMemoryLimitPercent =\n        jobConf.getFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT,\n            DEFAULT_SHUFFLE_MEMORY_LIMIT_PERCENT);\n    if (singleShuffleMemoryLimitPercent < 0.0f\n        || singleShuffleMemoryLimitPercent > 1.0f) {\n      throw new IllegalArgumentException(\"Invalid value for \"\n          + MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT + \": \"\n          + singleShuffleMemoryLimitPercent);\n    }\n\n    usedMemory = 0L;\n    commitMemory = 0L;\n    long maxSingleShuffleLimitConfiged =\n        (long)(memoryLimit * singleShuffleMemoryLimitPercent);\n    if(maxSingleShuffleLimitConfiged > Integer.MAX_VALUE) {\n      maxSingleShuffleLimitConfiged = Integer.MAX_VALUE;\n      LOG.info(\"The max number of bytes for a single in-memory shuffle cannot\" +\n          \" be larger than Integer.MAX_VALUE. Setting it to Integer.MAX_VALUE\");\n    }\n    this.maxSingleShuffleLimit = maxSingleShuffleLimitConfiged;\n    this.memToMemMergeOutputsThreshold =\n        jobConf.getInt(MRJobConfig.REDUCE_MEMTOMEM_THRESHOLD, ioSortFactor);\n    this.mergeThreshold = (long)(this.memoryLimit * \n                          jobConf.getFloat(","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java#L169-L205","documentation":"MergeManagerImpl's constructor validates mapreduce.reduce.shuffle.memory.limit.percent (maximum fraction of the shuffle memory that a single map output may occupy, default 0.25f) within [0.0, 1.0], throwing IllegalArgumentException otherwise. Map outputs larger than the resulting maxSingleShuffleLimit are copied straight to disk instead of into memory.","triggerScenarios":"The job conf sets mapreduce.reduce.shuffle.memory.limit.percent to a value below 0.0 or above 1.0 (e.g. '-1' or '25').","commonSituations":"Tuning attempts that enter a percent where a fraction is required; disabling in-memory shuffle with '-1' (the correct disable value is 0.0).","solutions":["Set mapreduce.reduce.shuffle.memory.limit.percent to a fraction in [0.0, 1.0] (default 0.25).","Use 0.0, not a negative number, to force all shuffle outputs to disk.","Keep it below mapreduce.reduce.shuffle.merge.percent (default 0.66) or construction fails with the maxSingleShuffleLimit check."],"exampleFix":"<!-- before -->\n<property><name>mapreduce.reduce.shuffle.memory.limit.percent</name><value>-1</value></property>\n<!-- after: 0 keeps every map output on disk -->\n<property><name>mapreduce.reduce.shuffle.memory.limit.percent</name><value>0.0</value></property>","handlingStrategy":"validation","validationCode":"// Validate before job submission\nfloat p = conf.getFloat(\"mapreduce.reduce.shuffle.memory.limit.percent\", 0.25f);\nif (p < 0.0f || p > 1.0f) { throw new IllegalArgumentException(\"mapreduce.reduce.shuffle.memory.limit.percent must be a fraction in [0,1]: \" + p); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use 0.0 (not a negative number) to disable in-memory shuffle copies.","Keep the value below mapreduce.reduce.shuffle.merge.percent to satisfy the follow-on config check."],"tags":["hadoop","mapreduce","shuffle","config","memory","validation"],"backgroundTag":"config-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}