{"record":{"id":"9e7ba114bc0faaa0","repo":"apache/hadoop","slug":"too-many-chunks-created-with-splitratio-splitrati","errorCode":null,"errorMessage":"Too many chunks created with splitRatio:{splitRatio}, numMaps:{numMaps}. Reduce numMaps or decrease split-ratio to proceed.","messagePattern":"Too many chunks created with splitRatio:(.+?), numMaps:(.+?)\\. Reduce numMaps or decrease split-ratio to proceed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java","lineNumber":185,"sourceCode":"        // Shuffle into open chunks.\n        openChunks.get(recordCounter%nChunksOpenAtOnce).write(relPath, fileStatus);\n        ++recordCounter;\n      }\n\n    } finally {\n      closeAll(openChunks);\n      chunksFinal.addAll(openChunks);\n      IOUtils.closeStream(reader);\n    }\n\n    LOG.info(\"Number of dynamic-chunk-files created: \" + chunksFinal.size()); \n    return chunksFinal;\n  }\n\n  private static void validateNumChunksUsing(int splitRatio, int numMaps,\n      int maxChunksTolerable) throws IOException {\n    if (splitRatio * numMaps > maxChunksTolerable)\n      throw new IOException(\"Too many chunks created with splitRatio:\"\n                 + splitRatio + \", numMaps:\" + numMaps\n                 + \". Reduce numMaps or decrease split-ratio to proceed.\");\n  }\n\n  private static void closeAll(List<DynamicInputChunk> chunks) {\n    for (DynamicInputChunk chunk: chunks)\n      chunk.close();\n  }\n\n  private List<DynamicInputChunk> createChunks(int chunkCount,\n      int nChunksTotal, int nChunksOpenAtOnce)\n      throws IOException {\n    List<DynamicInputChunk> chunks = new ArrayList<DynamicInputChunk>();\n    int chunkIdUpperBound\n            = Math.min(nChunksTotal, chunkCount + nChunksOpenAtOnce);\n\n    // If there will be fewer than nChunksOpenAtOnce chunks left after\n    // the current batch of chunks, fold the remaining chunks into","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java#L167-L203","documentation":"With the dynamic input strategy, DistCp caps the chunk-file count: validateNumChunksUsing() throws when splitRatio * numMaps exceeds distcp.dynamic.max.chunks.tolerable (default 400; split ratio default 2 via distcp.dynamic.split.ratio; numMaps comes from -m). It is a fail-fast check run right after the listing is created, before any data is copied.","triggerScenarios":"'-m 300' with the default split ratio 2 yields 600 > 400; distcp.dynamic.split.ratio raised to a large value; distcp.dynamic.max.chunks.tolerable lowered below numMaps * splitRatio.","commonSituations":"Scaling up -m for large migrations without adjusting the chunk cap; tuning split-ratio to balance very uneven file sizes; copying huge listings with many maps.","solutions":["Reduce the map count (e.g. -m 100) so splitRatio * numMaps stays within the cap","Lower the split ratio: -Ddistcp.dynamic.split.ratio=1","Raise the cap if the chunk-file overhead is acceptable: -Ddistcp.dynamic.max.chunks.tolerable=<n>","Or switch to -strategy uniform, which does not chunk the listing"],"exampleFix":"# before: 300 maps * default split-ratio 2 = 600 > 400 chunks\nhadoop distcp -m 300 -strategy dynamic hdfs://src hdfs://dst\n\n# after (either)\nhadoop distcp -m 100 -strategy dynamic hdfs://src hdfs://dst\nhadoop distcp -m 300 -Ddistcp.dynamic.split.ratio=1 hdfs://src hdfs://dst","handlingStrategy":"validation","validationCode":"int splitRatio = conf.getInt(\"distcp.dynamic.split.ratio\", 2);\nint maxChunks  = conf.getInt(\"distcp.dynamic.max.chunks.tolerable\", 400);\nif (splitRatio * numMaps > maxChunks) {\n  throw new IllegalArgumentException(\"lower -m to <= \" + (maxChunks / splitRatio)\n      + \" or reduce distcp.dynamic.split.ratio before submit\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat -m, distcp.dynamic.split.ratio and the chunk cap as one budget: splitRatio * numMaps <= 400","Default ratio is 2, so keep -m under 200 with defaults","Consider -strategy uniform when the dynamic cap is limiting"],"tags":["distcp","hadoop","dynamic","configuration","capacity"],"backgroundTag":"config-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}