{"record":{"id":"4addebe042ac2560","repo":"apache/hadoop","slug":"blocksperchunk-is-invalid-chunksizestr","errorCode":null,"errorMessage":"blocksPerChunk is invalid: {chunkSizeStr}","messagePattern":"blocksPerChunk is invalid: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java","lineNumber":225,"sourceCode":"      LOG.warn(DistCpOptionSwitch.FILE_LIMIT.getSwitch() + \" is a deprecated\" +\n          \" option. Ignoring.\");\n    }\n\n    if (command.hasOption(DistCpOptionSwitch.SIZE_LIMIT.getSwitch())) {\n      LOG.warn(DistCpOptionSwitch.SIZE_LIMIT.getSwitch() + \" is a deprecated\" +\n          \" option. Ignoring.\");\n    }\n\n    if (command.hasOption(DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch())) {\n      final String chunkSizeStr = getVal(command,\n          DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch().trim());\n      try {\n        int csize = Integer.parseInt(chunkSizeStr);\n        csize = csize > 0 ? csize : 0;\n        LOG.info(\"Set distcp blocksPerChunk to \" + csize);\n        builder.withBlocksPerChunk(csize);\n      } catch (NumberFormatException e) {\n        throw new IllegalArgumentException(\"blocksPerChunk is invalid: \"\n            + chunkSizeStr, e);\n      }\n    }\n\n    if (command.hasOption(DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch())) {\n      final String copyBufferSizeStr = getVal(command,\n          DistCpOptionSwitch.COPY_BUFFER_SIZE.getSwitch().trim());\n      try {\n        int copyBufferSize = Integer.parseInt(copyBufferSizeStr);\n        builder.withCopyBufferSize(copyBufferSize);\n      } catch (NumberFormatException e) {\n        throw new IllegalArgumentException(\"copyBufferSize is invalid: \"\n            + copyBufferSizeStr, e);\n      }\n    }\n\n    return builder.build();\n  }","sourceCodeStart":207,"sourceCodeEnd":243,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java#L207-L243","documentation":"The -blocksPerChunk option value must parse as a Java int; it controls splitting large files into N-block chunks copied in parallel and concatenated at commit. Only non-numeric text throws - note that a value <= 0 parses fine and is silently clamped to 0, which disables chunking (you only see 'Set distcp blocksPerChunk to 0' in the log). So this error means the value was not an integer at all.","triggerScenarios":"-blocksPerChunk 8m (assuming a size unit), -blocksPerChunk 2.5, or a non-numeric placeholder from a script.","commonSituations":"operators assuming the flag takes a byte size with a unit suffix, when it takes a count of HDFS blocks per chunk; templated values carrying units; documentation from other tools (e.g. -copyBufferSize in bytes) causing format confusion.","solutions":["Pass a plain unit-less integer block count, e.g. -blocksPerChunk 8.","Remove any size suffix (k/m/g) - the value is 'blocks per chunk', not bytes.","After running, grep the client log for 'Set distcp blocksPerChunk to' to confirm the effective value was not clamped to 0."],"exampleFix":"# before: '8m' is not an integer\nhadoop distcp -blocksPerChunk 8m hdfs://nn/src hdfs://nn/tgt\n\n# after: 8 HDFS blocks per chunk file\nhadoop distcp -blocksPerChunk 8 hdfs://nn/src hdfs://nn/tgt","handlingStrategy":"validation","validationCode":"// Wrapper validation: -blocksPerChunk is a block COUNT, unit-less\nString v = args[i + 1];\nif (!v.matches(\"[0-9]+\")) {\n  throw new IllegalArgumentException(\n      \"-blocksPerChunk must be an integer block count, got: \" + v);\n}\nif (Integer.parseInt(v) <= 0) {\n  LOG.warn(\"-blocksPerChunk \" + v + \" disables chunking (clamped to 0)\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  OptionsParser.parse(args);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"blocksPerChunk is invalid\")) {\n    throw new IllegalArgumentException(\n        \"-blocksPerChunk takes a block count like 8, not a size\", e);\n  }\n  throw e;\n}","preventionTips":["Remember -blocksPerChunk counts HDFS blocks per chunk - never append size units.","After launch, confirm the log line 'Set distcp blocksPerChunk to N' shows the value you intended (<= 0 silently disables).","Only enable chunking between FileSystems that support concat (HDFS-to-HDFS)."],"tags":["distcp","cli","blocksperchunk","number-format","invalid-arguments","chunking"],"backgroundTag":"invalid-numeric-option","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}