apache/hadoop · error · PathExistsException

File exists

Error message

File exists

What it means

Thrown by CommandWithDestination.processArguments() in the single-source case: the destination exists, is NOT a directory, and no -f (overwrite) flag was set. PathExistsException renders as 'dst: File exists'. The shell deliberately refuses to clobber an existing file without explicit consent.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java:238

    }
  }

  @Override
  protected void processArguments(LinkedList<PathData> args)
  throws IOException {
    // if more than one arg, the destination must be a directory
    // if one arg, the dst must not exist or must be a directory
    if (args.size() > 1) {
      if (!dst.exists) {
        throw new PathNotFoundException(dst.toString());
      }
      if (!dst.stat.isDirectory()) {
        throw new PathIsNotDirectoryException(dst.toString());
      }
    } else if (dst.exists) {
      if (!dst.stat.isDirectory() && !overwrite) {
        LOG.debug("Destination file exists: {}", dst.stat);
        throw new PathExistsException(dst.toString());
      }
    } else if (!dst.parentExists()) {
      throw new PathNotFoundException(dst.toString())
          .withFullyQualifiedPath(dst.path.toUri().toString());
    }
    super.processArguments(args);
  }

  @Override
  protected void processPathArgument(PathData src)
  throws IOException {
    if (src.stat.isDirectory() && src.fs.equals(dst.fs)) {
      PathData target = getTargetPath(src);
      String srcPath = src.fs.makeQualified(src.path).toString();
      String dstPath = dst.fs.makeQualified(target.path).toString();
      if (dstPath.equals(srcPath)) {
        PathIOException e = new PathIOException(src.toString(),
            "are identical");

View on GitHub (pinned to 2add963021)

Solutions

1. Add the overwrite flag: 'hdfs dfs -put -f new.csv /warehouse/data.csv'
2. If the existing file must be kept, choose a new destination name (e.g. timestamped)
3. Delete or archive the old file first ('hdfs dfs -rm') when -f is unavailable for the command variant in use

Example fix

# before
hdfs dfs -put new.csv /warehouse/data.csv
# after
hdfs dfs -put -f new.csv /warehouse/data.csv

// single-source copy: decide overwrite policy up front
boolean overwriteWanted = true;
if (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory() && !overwriteWanted) {
  throw new FileAlreadyExistsException("refusing to overwrite existing file " + dst);
}

try {
  shellRun("-put", src, dst);
} catch (PathExistsException e) {
  // existing file at destination and no -f given; retry with explicit overwrite
  shellRun("-put", "-f", src, dst);
}

When it happens

Trigger: 'hdfs dfs -put new.csv /warehouse/data.csv' when data.csv already exists and -f omitted; re-running a failed pipeline step whose output file was already written; -cp onto an existing file without -f.

Common situations: Job retries and backfill scripts hitting their own previous partial output; cron jobs assuming overwrite semantics like GNU cp -f; developers expecting put to truncate like '>' in shell redirection.

Related errors

• No such file or directory
• Too many matches
• Is not a directory
• Operation not supported
• Is a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/69d4a63b42ce1622. Report an issue: GitHub.