apache/hadoop · error · UnsupportedOperationException

blocksperchunk is not supported since the target file system

Error message

blocksperchunk is not supported since the target file system doesn't support concat.

What it means

Before accepting -blocksperchunk, DistCp.checkSplitLargeFile probes the target filesystem by invoking targetFS.concat(...) — the argument values are irrelevant (null/null); what matters is whether the call throws UnsupportedOperationException, which is the FileSystem.concat default. If it does, DistCp rethrows UnsupportedOperationException with this message: chunked copies are reassembled on the target via concat, so a target without concat cannot support chunking.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:290

        targetExists);
  }

  /**
   * Check splitting large files is supported and populate configs.
   */
  private void checkSplitLargeFile() throws IOException {
    if (!context.splitLargeFile()) {
      return;
    }

    final Path target = context.getTargetPath();
    final FileSystem targetFS = target.getFileSystem(getConf());
    try {
      Path[] src = null;
      Path tgt = null;
      targetFS.concat(tgt, src);
    } catch (UnsupportedOperationException use) {
      throw new UnsupportedOperationException(
          DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch() +
              " is not supported since the target file system doesn't" +
              " support concat.", use);
    } catch (Exception e) {
      // Ignore other exception
    }

    LOG.info("Set " +
        DistCpConstants.CONF_LABEL_SIMPLE_LISTING_RANDOMIZE_FILES
        + " to false since " + DistCpOptionSwitch.BLOCKS_PER_CHUNK.getSwitch()
        + " is passed.");
    getConf().setBoolean(
        DistCpConstants.CONF_LABEL_SIMPLE_LISTING_RANDOMIZE_FILES, false);
  }

  /**
   * Create Job object for submitting it, with all the configuration
   *

View on GitHub (pinned to 2add963021)

Solutions

1. Drop -blocksperchunk when the target is not HDFS
2. Or stage chunked to HDFS first, then move to the object store (distcp -blocksperchunk src -> hdfs://stage, then distcp/cp stage -> object store)
3. If you maintain the connector, implement FileSystem.concat so chunks can be reassembled
4. Probe with a single-file chunked copy before launching a production job

Example fix

# before
hadoop distcp -blocksperchunk 256 hdfs://src s3a://bucket/dst
# -> -blocksperchunk is not supported since the target file system doesn't support concat.

# after: no chunking for object stores
hadoop distcp hdfs://src s3a://bucket/dst

FileSystem tfs = targetPath.getFileSystem(conf);
boolean concatSupported;
try {
  tfs.concat(null, null); // same probe DistCp uses; args are irrelevant
  concatSupported = true;
} catch (UnsupportedOperationException e) {
  concatSupported = false;
} catch (Exception ignored) {
  concatSupported = true; // FS implemented concat and rejected the null args
}
if (useBlocksPerChunk && !concatSupported) {
  throw new UnsupportedOperationException(
      "target " + tfs.getUri() + " does not support concat; "
      + "drop -blocksperchunk or target HDFS");
}

When it happens

Trigger: Running hadoop distcp -blocksperchunk N with the destination on a filesystem that does not implement concat: file:// (local), object stores (s3a://, abfs://, gs://), or any FileSystem that did not override concat.

Common situations: Chunking flags copied from an HDFS-to-HDFS runbook into object-store copies; targeting local FS in tests; older connector versions lacking concat support.

Related errors

• Failed to concat chunk files for {targetFile}
• Inconsistent sequence file: current chunk file {srcFileStatu
• Fail to rename tmp file (={tmp}) to destination file (={dst}
• File " + lastFileStatus.getPath() + " " + lastChunkOffset +
• blocksPerChunk is invalid: {chunkSizeStr}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d443594d74255132. Report an issue: GitHub.