apache/hadoop · error · IllegalArgumentException

blocksPerChunk is invalid: {chunkSizeStr}

Error message

blocksPerChunk is invalid: {chunkSizeStr}

What it means

The -blocksPerChunk option value must parse as a Java int; it controls splitting large files into N-block chunks copied in parallel and concatenated at commit. Only non-numeric text throws - note that a value <= 0 parses fine and is silently clamped to 0, which disables chunking (you only see 'Set distcp blocksPerChunk to 0' in the log). So this error means the value was not an integer at all.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:225

      LOG.warn(DistCpOptionSwitch.FILE_LIMIT.getSwitch() + " is a deprecated" +
          " option. Ignoring.");
    }

    if (command.hasOption(DistCpOptionSwitch.SIZE_LIMIT.getSwitch())) {
      LOG.warn(DistCpOptionSwitch.SIZE_LIMIT.getSwitch() + " is a deprecated" +
          " option. Ignoring.");
    }

    if (command.hasOption(DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch())) {
      final String chunkSizeStr = getVal(command,
          DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch().trim());
      try {
        int csize = Integer.parseInt(chunkSizeStr);
        csize = csize > 0 ? csize : 0;
        LOG.info("Set distcp blocksPerChunk to " + csize);
        builder.withBlocksPerChunk(csize);
      } catch (NumberFormatException e) {
        throw new IllegalArgumentException("blocksPerChunk is invalid: "
            + chunkSizeStr, e);
      }
    }

    if (command.hasOption(DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch())) {
      final String copyBufferSizeStr = getVal(command,
          DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch().trim());
      try {
        int copyBufferSize = Integer.parseInt(copyBufferSizeStr);
        builder.withCopyBufferSize(copyBufferSize);
      } catch (NumberFormatException e) {
        throw new IllegalArgumentException("copyBufferSize is invalid: "
            + copyBufferSizeStr, e);
      }
    }

    return builder.build();
  }

View on GitHub (pinned to 2add963021)

Solutions

  1. Pass a plain unit-less integer block count, e.g. -blocksPerChunk 8.
  2. Remove any size suffix (k/m/g) - the value is 'blocks per chunk', not bytes.
  3. After running, grep the client log for 'Set distcp blocksPerChunk to' to confirm the effective value was not clamped to 0.

Example fix

# before: '8m' is not an integer
hadoop distcp -blocksPerChunk 8m hdfs://nn/src hdfs://nn/tgt

# after: 8 HDFS blocks per chunk file
hadoop distcp -blocksPerChunk 8 hdfs://nn/src hdfs://nn/tgt
Defensive patterns

Strategy: validation

Validate before calling

// Wrapper validation: -blocksPerChunk is a block COUNT, unit-less
String v = args[i + 1];
if (!v.matches("[0-9]+")) {
  throw new IllegalArgumentException(
      "-blocksPerChunk must be an integer block count, got: " + v);
}
if (Integer.parseInt(v) <= 0) {
  LOG.warn("-blocksPerChunk " + v + " disables chunking (clamped to 0)");
}

Try / catch

try {
  OptionsParser.parse(args);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("blocksPerChunk is invalid")) {
    throw new IllegalArgumentException(
        "-blocksPerChunk takes a block count like 8, not a size", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: -blocksPerChunk 8m (assuming a size unit), -blocksPerChunk 2.5, or a non-numeric placeholder from a script.

Common situations: operators assuming the flag takes a byte size with a unit suffix, when it takes a count of HDFS blocks per chunk; templated values carrying units; documentation from other tools (e.g. -copyBufferSize in bytes) causing format confusion.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4addebe042ac2560. Report an issue: GitHub.