apache/hadoop · error · IOException

Too many chunks created with splitRatio:{splitRatio}, numMap

Error message

Too many chunks created with splitRatio:{splitRatio}, numMaps:{numMaps}. Reduce numMaps or decrease split-ratio to proceed.

What it means

With the dynamic input strategy, DistCp caps the chunk-file count: validateNumChunksUsing() throws when splitRatio * numMaps exceeds distcp.dynamic.max.chunks.tolerable (default 400; split ratio default 2 via distcp.dynamic.split.ratio; numMaps comes from -m). It is a fail-fast check run right after the listing is created, before any data is copied.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java:185

        // Shuffle into open chunks.
        openChunks.get(recordCounter%nChunksOpenAtOnce).write(relPath, fileStatus);
        ++recordCounter;
      }

    } finally {
      closeAll(openChunks);
      chunksFinal.addAll(openChunks);
      IOUtils.closeStream(reader);
    }

    LOG.info("Number of dynamic-chunk-files created: " + chunksFinal.size()); 
    return chunksFinal;
  }

  private static void validateNumChunksUsing(int splitRatio, int numMaps,
      int maxChunksTolerable) throws IOException {
    if (splitRatio * numMaps > maxChunksTolerable)
      throw new IOException("Too many chunks created with splitRatio:"
                 + splitRatio + ", numMaps:" + numMaps
                 + ". Reduce numMaps or decrease split-ratio to proceed.");
  }

  private static void closeAll(List<DynamicInputChunk> chunks) {
    for (DynamicInputChunk chunk: chunks)
      chunk.close();
  }

  private List<DynamicInputChunk> createChunks(int chunkCount,
      int nChunksTotal, int nChunksOpenAtOnce)
      throws IOException {
    List<DynamicInputChunk> chunks = new ArrayList<DynamicInputChunk>();
    int chunkIdUpperBound
            = Math.min(nChunksTotal, chunkCount + nChunksOpenAtOnce);

    // If there will be fewer than nChunksOpenAtOnce chunks left after
    // the current batch of chunks, fold the remaining chunks into

View on GitHub (pinned to 2add963021)

Solutions

1. Reduce the map count (e.g. -m 100) so splitRatio * numMaps stays within the cap
2. Lower the split ratio: -Ddistcp.dynamic.split.ratio=1
3. Raise the cap if the chunk-file overhead is acceptable: -Ddistcp.dynamic.max.chunks.tolerable=<n>
4. Or switch to -strategy uniform, which does not chunk the listing

Example fix

# before: 300 maps * default split-ratio 2 = 600 > 400 chunks
hadoop distcp -m 300 -strategy dynamic hdfs://src hdfs://dst

# after (either)
hadoop distcp -m 100 -strategy dynamic hdfs://src hdfs://dst
hadoop distcp -m 300 -Ddistcp.dynamic.split.ratio=1 hdfs://src hdfs://dst

int splitRatio = conf.getInt("distcp.dynamic.split.ratio", 2);
int maxChunks  = conf.getInt("distcp.dynamic.max.chunks.tolerable", 400);
if (splitRatio * numMaps > maxChunks) {
  throw new IllegalArgumentException("lower -m to <= " + (maxChunks / splitRatio)
      + " or reduce distcp.dynamic.split.ratio before submit");
}

When it happens

Trigger: '-m 300' with the default split ratio 2 yields 600 > 400; distcp.dynamic.split.ratio raised to a large value; distcp.dynamic.max.chunks.tolerable lowered below numMaps * splitRatio.

Common situations: Scaling up -m for large migrations without adjusting the chunk cap; tuning split-ratio to balance very uneven file sizes; copying huge listings with many maps.

Related errors

• Commit directory not configured
• Working directory not configured
• Unable to release chunk at path: {chunkFilePath}
• No more entry in " + f
• f + " is a directory"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9e7ba114bc0faaa0. Report an issue: GitHub.