apache/hadoop · error · PathIsNotEmptyDirectoryException

Directory is not empty

Error message

Directory is not empty

What it means

PathIsNotEmptyDirectoryException ('Directory is not empty') thrown by Rmdir.processPath (Delete.java:215): 'hadoop fs -rmdir' was called on a directory whose listStatus() returned entries, and the -ignore-fail-on-non-empty flag was not set. rmdir intentionally refuses to delete non-empty directories, POSIX style.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java:215

    @Override
    protected void processOptions(LinkedList<String> args) throws IOException {
      CommandFormat cf = new CommandFormat(
          1, Integer.MAX_VALUE, "-ignore-fail-on-non-empty");
      cf.parse(args);
      ignoreNonEmpty = cf.getOpt("-ignore-fail-on-non-empty");
    }

    @Override
    protected void processPath(PathData item) throws IOException {
      if (!item.stat.isDirectory()) {
        throw new PathIsNotDirectoryException(item.toString());
      }      
      if (item.fs.listStatus(item.path).length == 0) {
        if (!item.fs.delete(item.path, false)) {
          throw new PathIOException(item.toString());
        }
      } else if (!ignoreNonEmpty) {
        throw new PathIsNotEmptyDirectoryException(item.toString());
      }
    }
  }

  // delete files from the trash that are older
  // than the retention threshold.
  static class Expunge extends FsCommand {
    private static final String OPTION_FILESYSTEM = "fs";

    public static final String NAME = "expunge";
    public static final String USAGE =
        "[-immediate] [-" + OPTION_FILESYSTEM + " <path>]";
    public static final String DESCRIPTION =
        "Delete files from the trash that are older " +
            "than the retention threshold";

    private boolean emptyImmediately = false;
    private String fsArgument;

View on GitHub (pinned to 2add963021)

Solutions

1. If recursive removal is fine: 'hadoop fs -rm -r /data/dir'
2. If you want best-effort rmdir semantics, add the flag: 'hadoop fs -rmdir -ignore-fail-on-non-empty /data/dir'
3. Empty the directory first ('hadoop fs -rm "/data/dir/*"') then rmdir — beware subdirectories need -r too
4. List the directory ('hdfs dfs -ls -a /data/dir') to find what is keeping it non-empty

Example fix

# before
hadoop fs -rmdir /output/job1     # Directory is not empty

# after
hadoop fs -rm -r /output/job1

if (fs.listStatus(path).length != 0) {
  // decide: fs.delete(path, true) OR throw PathIsNotEmptyDirectoryException
}

static boolean isEmptyDirectory(FileSystem fs, Path p) throws IOException {
  return fs.getFileStatus(p).isDirectory() && fs.listStatus(p).length == 0;
}

try {
  fs.delete(path, false);
} catch (PathIsNotEmptyDirectoryException e) {
  // escalate to recursive delete only if data loss is acceptable
}

When it happens

Trigger: 'hadoop fs -rmdir /data/dir' when /data/dir still contains files or subdirectories; hidden or zero-length files (e.g. .crc-style or _temporary) left by failed jobs; scripts that rmdir immediately after deleting visible files while background tasks recreate entries.

Common situations: Post-job cleanup after a partially failed job leaves _temporary/ or part files; users expecting rmdir to behave like rm -r; testing cleanup logic against directories with dot-files.

Related errors

• Is not a directory
• Can not delete the directory: [%s], as it is not empty and o
• Illegal option {}
• Not enough arguments: expected {} but got {}
• Too many arguments: expected {} but got {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6c35d1d01ee6da73. Report an issue: GitHub.