apache/hadoop · error · InvalidInputException

Multiple source being copied to a file: {targetPath}

Error message

Multiple source being copied to a file: {targetPath}

What it means

SimpleCopyListing detected that the target path exists and is a regular file. A file target can receive exactly one source file; with multiple source paths (after glob expansion, which happens first) there is nowhere to put the remaining files, so listing aborts before job submission.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java:161

    Path targetPath = context.getTargetPath();
    FileSystem targetFS = targetPath.getFileSystem(getConf());
    boolean targetExists = false;
    boolean targetIsFile = false;
    try {
      targetIsFile = targetFS.getFileStatus(targetPath).isFile();
      targetExists = true;
    } catch (FileNotFoundException ignored) {
    }
    targetPath = targetFS.makeQualified(targetPath);
    final boolean targetIsReservedRaw =
        Path.getPathWithoutSchemeAndAuthority(targetPath).toString().
            startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME);

    //If target is a file, then source has to be single file
    if (targetIsFile) {
      if (context.getSourcePaths().size() > 1) {
        throw new InvalidInputException("Multiple source being copied to a file: " +
            targetPath);
      }

      Path srcPath = context.getSourcePaths().get(0);
      FileSystem sourceFS = srcPath.getFileSystem(getConf());
      if (!sourceFS.isFile(srcPath)) {
        throw new InvalidInputException("Cannot copy " + srcPath +
            ", which is not a file to " + targetPath);
      }
    }

    if (context.shouldAtomicCommit() && targetExists) {
      throw new InvalidInputException("Target path for atomic-commit already exists: " +
        targetPath + ". Cannot atomic-commit to pre-existing target-path.");
    }

    for (Path path: context.getSourcePaths()) {
      FileSystem fs = path.getFileSystem(getConf());

View on GitHub (pinned to 2add963021)

Solutions

1. Make the target a directory: hadoop fs -rm <targetFile> && hadoop fs -mkdir <targetDir>, then re-run with the directory as target.
2. Or reduce the inputs to exactly one source file when the file target is intended.
3. Check what the target really is first: hadoop fs -ls <target> (isFile() on the target decided this).

Example fix

# before: target exists as a file but two inputs are given
hadoop distcp -overwrite hdfs://a/f1 hdfs://a/f2 hdfs://b/t

# after: target is a directory
hadoop fs -rm hdfs://b/t && hadoop fs -mkdir hdfs://b/t
hadoop distcp -overwrite hdfs://a/f1 hdfs://a/f2 hdfs://b/t

// Pre-flight: a file target may receive exactly one source file
FileSystem tfs = targetPath.getFileSystem(conf);
boolean targetIsFile = tfs.exists(targetPath) && tfs.getFileStatus(targetPath).isFile();
int nSources = context.getSourcePaths().size(); // post-glob count if you expand first
if (targetIsFile && nSources > 1) {
  throw new InvalidInputException(
      "Target " + targetPath + " is a file but " + nSources + " sources given");
}

try {
  copyListing.doBuildListing(listingFile, context);
} catch (InvalidInputException e) {
  if (e.getMessage().startsWith("Multiple source being copied to a file")) {
    // remove/rename the file target or cut inputs to one, then rebuild listing
    fixTargetTypeAndRerun(e.getMessage());
  } else {
    throw e;
  }
}

When it happens

Trigger: hadoop distcp hdfs://a/f1 hdfs://a/f2 hdfs://b/existingFile; a single glob source that expands to 2+ files with a pre-existing file target, e.g. hadoop distcp 'hdfs://a/*.txt' hdfs://b/file (GlobbedCopyListing expands first, then SimpleCopyListing sees multiple sourcePaths).

Common situations: assuming distcp merges/overwrites multiple files into one file target; a glob matching more than expected; a target that was meant to be a directory but was created as a file by an earlier step.

Related errors

• Cannot copy {srcPath}, which is not a file to {targetPath}
• Target path for atomic-commit already exists: {targetPath}.
• File " + lastFileStatus.getPath() + " and " + currentFileSta
• Nothing to process. Source paths::EMPTY
• {p} doesn't exist

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/863c50eb96e4b066. Report an issue: GitHub.