{"record":{"id":"e7021b5f57671cbc","repo":"apache/hadoop","slug":"invalid-configuration-maxsingleshufflelimit-shoul","errorCode":null,"errorMessage":"Invalid configuration: maxSingleShuffleLimit should be less than mergeThreshold maxSingleShuffleLimit: {maxSingleShuffleLimit}mergeThreshold: {mergeThreshold}","messagePattern":"Invalid configuration: maxSingleShuffleLimit should be less than mergeThreshold maxSingleShuffleLimit: (.+?)mergeThreshold: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java","lineNumber":215,"sourceCode":"      maxSingleShuffleLimitConfiged = Integer.MAX_VALUE;\n      LOG.info(\"The max number of bytes for a single in-memory shuffle cannot\" +\n          \" be larger than Integer.MAX_VALUE. Setting it to Integer.MAX_VALUE\");\n    }\n    this.maxSingleShuffleLimit = maxSingleShuffleLimitConfiged;\n    this.memToMemMergeOutputsThreshold =\n        jobConf.getInt(MRJobConfig.REDUCE_MEMTOMEM_THRESHOLD, ioSortFactor);\n    this.mergeThreshold = (long)(this.memoryLimit * \n                          jobConf.getFloat(\n                            MRJobConfig.SHUFFLE_MERGE_PERCENT,\n                            MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT));\n    LOG.info(\"MergerManager: memoryLimit=\" + memoryLimit + \", \" +\n             \"maxSingleShuffleLimit=\" + maxSingleShuffleLimit + \", \" +\n             \"mergeThreshold=\" + mergeThreshold + \", \" + \n             \"ioSortFactor=\" + ioSortFactor + \", \" +\n             \"memToMemMergeOutputsThreshold=\" + memToMemMergeOutputsThreshold);\n\n    if (this.maxSingleShuffleLimit >= this.mergeThreshold) {\n      throw new RuntimeException(\"Invalid configuration: \"\n          + \"maxSingleShuffleLimit should be less than mergeThreshold \"\n          + \"maxSingleShuffleLimit: \" + this.maxSingleShuffleLimit\n          + \"mergeThreshold: \" + this.mergeThreshold);\n    }\n\n    boolean allowMemToMemMerge = \n      jobConf.getBoolean(MRJobConfig.REDUCE_MEMTOMEM_ENABLED, false);\n    if (allowMemToMemMerge) {\n      this.memToMemMerger = \n        new IntermediateMemoryToMemoryMerger(this,\n                                             memToMemMergeOutputsThreshold);\n      this.memToMemMerger.start();\n    } else {\n      this.memToMemMerger = null;\n    }\n    \n    this.inMemoryMerger = createInMemoryMerger();\n    this.inMemoryMerger.start();","sourceCodeStart":197,"sourceCodeEnd":233,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java#L197-L233","documentation":"MergeManagerImpl derives mergeThreshold = memoryLimit * mapreduce.reduce.shuffle.merge.percent (default 0.66f) — the used-memory level that triggers a merge to free memory — and maxSingleShuffleLimit = memoryLimit * mapreduce.reduce.shuffle.memory.limit.percent (default 0.25f). If one shuffle could fill memory past the merge trigger, memory could never be reclaimed, so the constructor throws RuntimeException and the reduce task fails at startup.","triggerScenarios":"mapreduce.reduce.shuffle.memory.limit.percent >= mapreduce.reduce.shuffle.merge.percent, e.g. 0.8 with the default merge percent of 0.66; also reachable by raising merge percent below an existing limit percent.","commonSituations":"Aggressive tuning to keep large map outputs in memory without lowering the merge threshold correspondingly; config pairs copied from different tuning guides.","solutions":["Lower mapreduce.reduce.shuffle.memory.limit.percent below mapreduce.reduce.shuffle.merge.percent (defaults 0.25 < 0.66 always satisfy this).","Or raise mapreduce.reduce.shuffle.merge.percent toward (but above) the limit percent so merging triggers later.","If large outputs must stay in memory, increase reducer heap rather than the limit fraction."],"exampleFix":"// before: single-shuffle limit above the merge threshold -> RuntimeException\nconf.setFloat(\"mapreduce.reduce.shuffle.memory.limit.percent\", 0.8f); // merge.percent = 0.66\n// after: keep the limit strictly below the merge threshold\nconf.setFloat(\"mapreduce.reduce.shuffle.memory.limit.percent\", 0.5f); // < 0.66","handlingStrategy":"validation","validationCode":"// Validate the fraction pair before job submission\nfloat limit = conf.getFloat(\"mapreduce.reduce.shuffle.memory.limit.percent\", 0.25f);\nfloat merge = conf.getFloat(\"mapreduce.reduce.shuffle.merge.percent\", 0.66f);\nif (limit >= merge) { throw new IllegalArgumentException(\"memory.limit.percent (\" + limit + \") must stay below merge.percent (\" + merge + \")\"); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Tune shuffle-memory settings as a pair: single-shuffle limit strictly below the merge threshold.","Prefer raising reducer heap over pushing the limit fraction up."],"tags":["hadoop","mapreduce","shuffle","config","memory","merge","validation"],"backgroundTag":"config-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}