apache/hadoop · error · IllegalArgumentException

copyBufferSize is invalid: {copyBufferSizeStr}

Error message

copyBufferSize is invalid: {copyBufferSizeStr}

What it means

The -copyBufferSize option value must parse as a Java int specifying the copy buffer size in bytes. Integer.parseInt failed, so the value was not a plain integer - human-readable sizes like 64k or 1MB are the classic mistake.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:237

      try {
        int csize = Integer.parseInt(chunkSizeStr);
        csize = csize > 0 ? csize : 0;
        LOG.info("Set distcp blocksPerChunk to " + csize);
        builder.withBlocksPerChunk(csize);
      } catch (NumberFormatException e) {
        throw new IllegalArgumentException("blocksPerChunk is invalid: "
            + chunkSizeStr, e);
      }
    }

    if (command.hasOption(DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch())) {
      final String copyBufferSizeStr = getVal(command,
          DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch().trim());
      try {
        int copyBufferSize = Integer.parseInt(copyBufferSizeStr);
        builder.withCopyBufferSize(copyBufferSize);
      } catch (NumberFormatException e) {
        throw new IllegalArgumentException("copyBufferSize is invalid: "
            + copyBufferSizeStr, e);
      }
    }

    return builder.build();
  }

  /**
   * parseSourceAndTargetPaths is a helper method for parsing the source
   * and target paths.
   *
   * @param command command line arguments
   * @return        DistCpOptions
   */
  private static DistCpOptions.Builder parseSourceAndTargetPaths(
      CommandLine command) {
    Path targetPath;
    List<Path> sourcePaths = new ArrayList<Path>();

View on GitHub (pinned to 2add963021)

Solutions

1. Pass the byte count as a plain integer, e.g. -copyBufferSize 65536.
2. Convert units yourself: 64k = 65536, 1MB = 1048576.
3. Validate the value matches ^[0-9]+$ in the launcher script.

Example fix

# before
hadoop distcp -copyBufferSize 64k hdfs://nn/src hdfs://nn/tgt

# after: bytes only
hadoop distcp -copyBufferSize 65536 hdfs://nn/src hdfs://nn/tgt

// Wrapper validation: -copyBufferSize is a byte count, unit-less
String v = args[i + 1];
if (!v.matches("[0-9]+")) {
  throw new IllegalArgumentException(
      "-copyBufferSize must be bytes (e.g. 65536), got: " + v);
}

try {
  OptionsParser.parse(args);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("copyBufferSize is invalid")) {
    throw new IllegalArgumentException(
        "-copyBufferSize takes bytes (64k -> 65536), not suffixed sizes", e);
  }
  throw e;
}

When it happens

Trigger: -copyBufferSize 64k, -copyBufferSize 1MB, or a decimal value like 65536.0.

Common situations: operators porting settings from tools that accept size suffixes; runbooks written with human-readable units; unset variables yielding empty strings.

Related errors

• Bandwidth specified is invalid: {value}
• Number of liststatus threads is invalid: {value}
• Number of maps is invalid: {value}
• blocksPerChunk is invalid: {chunkSizeStr}
• Unable to parse arguments. {args}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/478533d53099caae. Report an issue: GitHub.