apache/hadoop · error · IOException

Cannot remove current dump directory: {}

Error message

Cannot remove current dump directory: {}

What it means

On startup RpcProgramNfs3.clearDirectory() wipes the NFS write dump directory (nfs.dump.dir, default /tmp/.hdfs-nfs) to start from a clean state. If FileUtil.fullyDelete() cannot remove the existing directory, startup fails with this IOException (a companion check then fails on mkdirs if creation is also impossible).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java:237

  }

  public static RpcProgramNfs3 createRpcProgramNfs3(NfsConfiguration config,
      DatagramSocket registrationSocket, boolean allowInsecurePorts)
      throws IOException {
    DefaultMetricsSystem.initialize("Nfs3");
    String displayName = DNS.getDefaultHost("default", "default")
        + config.getInt(NfsConfigKeys.DFS_NFS_SERVER_PORT_KEY,
            NfsConfigKeys.DFS_NFS_SERVER_PORT_DEFAULT);
    metrics = Nfs3Metrics.create(config, displayName);
    return new RpcProgramNfs3(config, registrationSocket, allowInsecurePorts);
  }
  
  private void clearDirectory(String writeDumpDir) throws IOException {
    File dumpDir = new File(writeDumpDir);
    if (dumpDir.exists()) {
      LOG.info("Delete current dump directory {}", writeDumpDir);
      if (!(FileUtil.fullyDelete(dumpDir))) {
        throw new IOException("Cannot remove current dump directory: "
            + dumpDir);
      }
    }
    LOG.info("Create new dump directory {}", writeDumpDir);
    if (!dumpDir.mkdirs()) {
      throw new IOException("Cannot create dump directory " + dumpDir);
    }
  }
  
  @Override
  public void startDaemons() {
    if (pauseMonitor == null) {
      pauseMonitor = new JvmPauseMonitor();
      pauseMonitor.init(config);
      pauseMonitor.start();
      metrics.getJvmMetrics().setPauseMonitor(pauseMonitor);
    }
    writeManager.startAsyncDataService();

View on GitHub (pinned to 2add963021)

Solutions

1. Manually remove the dump directory as a privileged user: rm -rf /tmp/.hdfs-nfs (or your nfs.dump.dir), then restart the gateway.
2. Configure nfs.dump.dir to a dedicated directory owned by the gateway user instead of /tmp.
3. Always run the NFS gateway under the same service account so dump files stay deletable.

Example fix

# before: gateway startup fails with 'Cannot remove current dump directory'
sudo rm -rf /tmp/.hdfs-nfs
# after (nfssite-site.xml + ownership fix)
<property><name>nfs.dump.dir</name><value>/var/lib/hadoop-nfs/dump</value></property>
chown -R hdfs:hdfs /var/lib/hadoop-nfs

/* pre-start check: dump dir must be removable by the gateway user */
File dumpDir = new File(conf.get("nfs.dump.dir", "/tmp/.hdfs-nfs"));
if (dumpDir.exists() && !Files.isWritable(dumpDir.getParentFile())) {
    throw new IOException("Cannot manage dump dir " + dumpDir + " — fix ownership");
}
for (File f : dumpDir.listFiles() != null ? dumpDir.listFiles() : new File[0]) {
    if (!Files.isWritable(f.toPath().getParent()) && !f.canWrite()) {
        throw new IOException("Undeletable leftover in dump dir: " + f);
    }
}

try {
    /* gateway start */
} catch (IOException e) {
    if (e.getMessage().contains("Cannot remove current dump directory")) {
        // deterministic: clear the dir as its owner (or privileged), e.g.
        //   sudo rm -rf /tmp/.hdfs-nfs
        // then restart. Do not blindly delete if other services share the path.
    } else {
        throw e;
    }
}

When it happens

Trigger: The dump directory contains files the gateway user cannot delete: leftovers owned by root or a previous different service account, a read-only mount, SELinux denials, or the path pointing at a file/symlink rather than a directory.

Common situations: Switching the user the NFS gateway runs as (old dump files keep the former owner); /tmp cleaner daemons partially deleting the directory leaving odd ownership; restoring gateway state from images/backups that preserved stale ownership.

Related errors

• Cannot create dump directory {}
• Permission denied: user=%s, path="%s":%s:%s:%s%s
• {} doesn't support modifyAclEntries
• {} doesn't support removeAclEntries
• {} doesn't support removeAcl

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e61d280b7d414c21. Report an issue: GitHub.