apache/hadoop · error · IOException

Failed to concat chunk files for {targetFile}

Error message

Failed to concat chunk files for {targetFile}

What it means

With -blocksPerChunk N, large files are copied as chunk files named <target>.____distcpSplit____<offset>.<length> and CopyCommitter.concatFileChunks merges each chunk set back into the final file at job commit. This error means concat threw an IOException that is not a FileNotFoundException. A FileNotFound is deliberately swallowed (it implies CopyMapper skipped that file), so any other IO problem - NameNode in safe mode, permission loss, FS lacking concat support (object stores), chunks deleted mid-job - is rethrown when failures are not ignored. With -i it is only logged as a warning and the job 'succeeds' with unconcatenated chunk files left behind.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:274

          LOG.debug("  add " + targetFileChunkPath + " to concat.");
        }
        allChunkPaths.add(targetFileChunkPath);
        if (srcFileStatus.getChunkOffset() + srcFileStatus.getChunkLength()
            == srcFileStatus.getLen()) {
          // This is the last chunk of the splits, consolidate allChunkPaths
          try {
            concatFileChunks(conf, srcFileStatus.getPath(), targetFile,
                allChunkPaths, srcFileStatus);
          } catch (IOException e) {
            // If the concat failed because a chunk file doesn't exist,
            // then we assume that the CopyMapper has skipped copying this
            // file, and we ignore the exception here.
            // If a chunk file should have been created but it was not, then
            // the CopyMapper would have failed.
            if (!isFileNotFoundException(e)) {
              String emsg = "Failed to concat chunk files for " + targetFile;
              if (!ignoreFailures) {
                throw new IOException(emsg, e);
              } else {
                LOG.warn(emsg, e);
              }
            }
          }
          allChunkPaths.clear();
          lastFileStatus = null;
        } else {
          if (lastFileStatus == null) {
            lastFileStatus = new CopyListingFileStatus(srcFileStatus);
          } else {
            // Two neighboring chunks have to be consecutive ones for the same
            // file, for them to be merged
            if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) ||
                srcFileStatus.getChunkOffset() !=
                (lastFileStatus.getChunkOffset() +
                lastFileStatus.getChunkLength())) {
              String emsg = "Inconsistent sequence file: current " +

View on GitHub (pinned to 2add963021)

Solutions

1. Read the chained cause: for safe mode run 'hdfs dfsadmin -safemode leave' (or wait it out), then re-run the same distcp command - already-copied chunks are skipped and concat is re-attempted.
2. If the target FileSystem does not support concat (object store), drop -blocksPerChunk and re-run.
3. Restore permissions/ownership on the target dir so the job user can rename/concat, then retry.
4. If -i was set, audit job logs for 'Failed to concat chunk files' warns, and re-run those files - the target still holds <file>.____distcpSplit____. chunks until a successful concat.
5. Delete stray partial chunk files for the affected target before retrying: hadoop fs -rm '<targetFile>.____distcpSplit____.*'.

Example fix

# before: chunked copy onto a FileSystem without concat support
hadoop distcp -blocksPerChunk 8 s3a://bucket/src hdfs://nn/tgt

# after: chunk only between HDFS endpoints that support concat
hadoop distcp -blocksPerChunk 8 hdfs://nn/src hdfs://nn/tgt

// Before enabling -blocksPerChunk, confirm the target FS is HDFS-like (supports concat)
FileSystem tfs = targetPath.getFileSystem(conf);
if (!(tfs instanceof DistributedFileSystem)) {
  throw new UnsupportedOperationException(
      "-blocksPerChunk needs a FileSystem supporting concat, got " + tfs.getUri());
}
// and confirm the NN is healthy before the commit phase gets there
((DistributedFileSystem) tfs).getDataNodeStats().getDatanodeReport(DatanodeReportType.LIVE);

try {
  job.waitForCompletion(true); // commit phase performs the concat
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to concat chunk files")) {
    IOException cause = (IOException) e.getCause();
    // act on the real cause: safe mode -> wait/leave; perms -> fix; unsupported -> drop -blocksPerChunk
    handleConcatFailure(cause);
    rerunSameDistcpCommand(); // copied chunks are skipped, concat re-attempted
  } else {
    throw e;
  }
}

When it happens

Trigger: -blocksPerChunk against a target FileSystem whose concat is unsupported (s3a://, abfs:// and similar); NameNode in safe mode or unreachable during the commit phase; another process deleting/renaming the .____distcpSplit____. chunk files between the map phase and commit; permission or ownership changes on the target directory; -i masking the failure so stray chunk files accumulate silently.

Common situations: large-file backups tuned with -blocksPerChunk for parallelism; commit coinciding with NN restart or safe mode; jobs combining -i with -blocksPerChunk without auditing logs; object-store targets used with options designed for HDFS-to-HDFS.

Related errors

• blocksperchunk is not supported since the target file system
• Inconsistent sequence file: current chunk file {srcFileStatu
• Fail to rename tmp file (={tmp}) to destination file (={dst}
• Target path not specified. <target path> <src path> <src pat
• The number of source paths is less than 2. <target path> <sr

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2e27e69207280c31. Report an issue: GitHub.