{"record":{"id":"d443594d74255132","repo":"apache/hadoop","slug":"blocksperchunk-is-not-supported-since-the-target-f","errorCode":null,"errorMessage":"blocksperchunk is not supported since the target file system doesn't support concat.","messagePattern":"blocksperchunk is not supported since the target file system doesn't support concat\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java","lineNumber":290,"sourceCode":"        targetExists);\n  }\n\n  /**\n   * Check splitting large files is supported and populate configs.\n   */\n  private void checkSplitLargeFile() throws IOException {\n    if (!context.splitLargeFile()) {\n      return;\n    }\n\n    final Path target = context.getTargetPath();\n    final FileSystem targetFS = target.getFileSystem(getConf());\n    try {\n      Path[] src = null;\n      Path tgt = null;\n      targetFS.concat(tgt, src);\n    } catch (UnsupportedOperationException use) {\n      throw new UnsupportedOperationException(\n          DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch() +\n              \" is not supported since the target file system doesn't\" +\n              \" support concat.\", use);\n    } catch (Exception e) {\n      // Ignore other exception\n    }\n\n    LOG.info(\"Set \" +\n        DistCpConstants.CONF_LABEL_SIMPLE_LISTING_RANDOMIZE_FILES\n        + \" to false since \" + DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch()\n        + \" is passed.\");\n    getConf().setBoolean(\n        DistCpConstants.CONF_LABEL_SIMPLE_LISTING_RANDOMIZE_FILES, false);\n  }\n\n  /**\n   * Create Job object for submitting it, with all the configuration\n   *","sourceCodeStart":272,"sourceCodeEnd":308,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L272-L308","documentation":"Before accepting -blocksperchunk, DistCp.checkSplitLargeFile probes the target filesystem by invoking targetFS.concat(...) — the argument values are irrelevant (null/null); what matters is whether the call throws UnsupportedOperationException, which is the FileSystem.concat default. If it does, DistCp rethrows UnsupportedOperationException with this message: chunked copies are reassembled on the target via concat, so a target without concat cannot support chunking.","triggerScenarios":"Running hadoop distcp -blocksperchunk N with the destination on a filesystem that does not implement concat: file:// (local), object stores (s3a://, abfs://, gs://), or any FileSystem that did not override concat.","commonSituations":"Chunking flags copied from an HDFS-to-HDFS runbook into object-store copies; targeting local FS in tests; older connector versions lacking concat support.","solutions":["Drop -blocksperchunk when the target is not HDFS","Or stage chunked to HDFS first, then move to the object store (distcp -blocksperchunk src -> hdfs://stage, then distcp/cp stage -> object store)","If you maintain the connector, implement FileSystem.concat so chunks can be reassembled","Probe with a single-file chunked copy before launching a production job"],"exampleFix":"# before\nhadoop distcp -blocksperchunk 256 hdfs://src s3a://bucket/dst\n# -> -blocksperchunk is not supported since the target file system doesn't support concat.\n\n# after: no chunking for object stores\nhadoop distcp hdfs://src s3a://bucket/dst","handlingStrategy":"validation","validationCode":"FileSystem tfs = targetPath.getFileSystem(conf);\nboolean concatSupported;\ntry {\n  tfs.concat(null, null); // same probe DistCp uses; args are irrelevant\n  concatSupported = true;\n} catch (UnsupportedOperationException e) {\n  concatSupported = false;\n} catch (Exception ignored) {\n  concatSupported = true; // FS implemented concat and rejected the null args\n}\nif (useBlocksPerChunk && !concatSupported) {\n  throw new UnsupportedOperationException(\n      \"target \" + tfs.getUri() + \" does not support concat; \"\n      + \"drop -blocksperchunk or target HDFS\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Reserve -blocksperchunk for HDFS destinations","Run a one-file chunked probe copy before a production job","For custom filesystems, implement concat before advertising chunked-distcp support"],"tags":["distcp","blocksperchunk","concat","object-store","filesystem-support"],"backgroundTag":"unsupported-filesystem-operation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}