apache/hadoop · error · IOException

Cannot remove current dump directory

Error message

Cannot remove current dump directory: {}

What it means

On startup RpcProgramNfs3.clearDirectory() wipes the NFS write dump directory (nfs.dump.dir, default /tmp/.hdfs-nfs) to start from a clean state. If FileUtil.fullyDelete() cannot remove the existing directory, startup fails with this IOException (a companion check then fails on mkdirs if creation is also impossible).

Solutions

1. Manually remove the dump directory as a privileged user: rm -rf /tmp/.hdfs-nfs (or your nfs.dump.dir), then restart the gateway.
2. Configure nfs.dump.dir to a dedicated directory owned by the gateway user instead of /tmp.
3. Always run the NFS gateway under the same service account so dump files stay deletable.

Example fix

# before: gateway startup fails with 'Cannot remove current dump directory'
sudo rm -rf /tmp/.hdfs-nfs
# after (nfssite-site.xml + ownership fix)
<property><name>nfs.dump.dir</name><value>/var/lib/hadoop-nfs/dump</value></property>
chown -R hdfs:hdfs /var/lib/hadoop-nfs

/* pre-start check: dump dir must be removable by the gateway user */
File dumpDir = new File(conf.get("nfs.dump.dir", "/tmp/.hdfs-nfs"));
if (dumpDir.exists() && !Files.isWritable(dumpDir.getParentFile())) {
    throw new IOException("Cannot manage dump dir " + dumpDir + " — fix ownership");
}
for (File f : dumpDir.listFiles() != null ? dumpDir.listFiles() : new File[0]) {
    if (!Files.isWritable(f.toPath().getParent()) && !f.canWrite()) {
        throw new IOException("Undeletable leftover in dump dir: " + f);
    }
}

try {
    /* gateway start */
} catch (IOException e) {
    if (e.getMessage().contains("Cannot remove current dump directory")) {
        // deterministic: clear the dir as its owner (or privileged), e.g.
        //   sudo rm -rf /tmp/.hdfs-nfs
        // then restart. Do not blindly delete if other services share the path.
    } else {
        throw e;
    }
}

When it happens

Trigger: The dump directory contains files the gateway user cannot delete: leftovers owned by root or a previous different service account, a read-only mount, SELinux denials, or the path pointing at a file/symlink rather than a directory.

Common situations: Switching the user the NFS gateway runs as (old dump files keep the former owner); /tmp cleaner daemons partially deleting the directory leaving odd ownership; restoring gateway state from images/backups that preserved stale ownership.

Related errors

• Cannot create dump directory
• cannot access checkpoint directory.
• Cannot create directory
• Cannot create directory
• Cannot remove directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e61d280b7d414c21. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java:237

  }

  public static RpcProgramNfs3 createRpcProgramNfs3(NfsConfiguration config,
      DatagramSocket registrationSocket, boolean allowInsecurePorts)
      throws IOException {
    DefaultMetricsSystem.initialize("Nfs3");
    String displayName = DNS.getDefaultHost("default", "default")
        + config.getInt(NfsConfigKeys.DFS_NFS_SERVER_PORT_KEY,
            NfsConfigKeys.DFS_NFS_SERVER_PORT_DEFAULT);
    metrics = Nfs3Metrics.create(config, displayName);
    return new RpcProgramNfs3(config, registrationSocket, allowInsecurePorts);
  }
  
  private void clearDirectory(String writeDumpDir) throws IOException {
    File dumpDir = new File(writeDumpDir);
    if (dumpDir.exists()) {
      LOG.info("Delete current dump directory {}", writeDumpDir);
      if (!(FileUtil.fullyDelete(dumpDir))) {
        throw new IOException("Cannot remove current dump directory: "
            + dumpDir);
      }
    }
    LOG.info("Create new dump directory {}", writeDumpDir);
    if (!dumpDir.mkdirs()) {
      throw new IOException("Cannot create dump directory " + dumpDir);
    }
  }
  
  @Override
  public void startDaemons() {
    if (pauseMonitor == null) {
      pauseMonitor = new JvmPauseMonitor();
      pauseMonitor.init(config);
      pauseMonitor.start();
      metrics.getJvmMetrics().setPauseMonitor(pauseMonitor);
    }
    writeManager.startAsyncDataService();

View on GitHub (pinned to 2add963021)