apache/hadoop · critical · IOException

Cannot create dump directory {}

Error message

Cannot create dump directory {}

What it means

Thrown by the HDFS NFS3 gateway during initialization when it cannot create (or recreate) its write-dump directory, the local staging area where WRITE RPC payloads received before COMMIT are persisted (nfs.file.dump=true). clearDirectory() first fully deletes the existing dump dir and then calls mkdirs(); if mkdirs() returns false the gateway throws this IOException and refuses to start, because async WRITE handling depends on that directory (nfs.dump.dir, default /tmp/.hdfs-nfs).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java:243

    String displayName = DNS.getDefaultHost("default", "default")
        + config.getInt(NfsConfigKeys.DFS_NFS_SERVER_PORT_KEY,
            NfsConfigKeys.DFS_NFS_SERVER_PORT_DEFAULT);
    metrics = Nfs3Metrics.create(config, displayName);
    return new RpcProgramNfs3(config, registrationSocket, allowInsecurePorts);
  }
  
  private void clearDirectory(String writeDumpDir) throws IOException {
    File dumpDir = new File(writeDumpDir);
    if (dumpDir.exists()) {
      LOG.info("Delete current dump directory {}", writeDumpDir);
      if (!(FileUtil.fullyDelete(dumpDir))) {
        throw new IOException("Cannot remove current dump directory: "
            + dumpDir);
      }
    }
    LOG.info("Create new dump directory {}", writeDumpDir);
    if (!dumpDir.mkdirs()) {
      throw new IOException("Cannot create dump directory " + dumpDir);
    }
  }
  
  @Override
  public void startDaemons() {
    if (pauseMonitor == null) {
      pauseMonitor = new JvmPauseMonitor();
      pauseMonitor.init(config);
      pauseMonitor.start();
      metrics.getJvmMetrics().setPauseMonitor(pauseMonitor);
    }
    writeManager.startAsyncDataService();
    try {
      infoServer.start();
    } catch (IOException e) {
      LOG.error("failed to start web server", e);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the path and permissions: ls -ld <nfs.dump.dir value> (default /tmp/.hdfs-nfs) and confirm the gateway user can create/delete entries in its parent
2. Remove the stale dump directory left by a previous run as the right user: rm -rf /tmp/.hdfs-nfs (or chown it back to the gateway user with sudo)
3. Point nfs.dump.dir in core-site.xml at a dedicated, always-writable location with free disk space and restart the NFS3 gateway
4. Rule out a plain file with the same name, SELinux denials, and a full/read-only filesystem on that mount

Example fix

<!-- before: default shared tmp path the gateway user cannot (re)create -->
<property>
  <name>nfs.dump.dir</name>
  <value>/tmp/.hdfs-nfs</value>
</property>
<!-- after: dedicated writable location, mkdir + chown nfs-user first -->
<property>
  <name>nfs.dump.dir</name>
  <value>/var/nfs3/dump</value>
</property>

// Pre-flight before starting the NFS3 gateway
import java.nio.file.*;

Path dumpDir = Paths.get(conf.get("nfs.dump.dir", "/tmp/.hdfs-nfs"));
try {
  if (Files.exists(dumpDir)) {
    try (var s = Files.list(dumpDir)) {
      for (Path p : (Iterable<Path>) s::iterator) Files.deleteIfExists(p);
    }
    Files.deleteIfExists(dumpDir);
  }
  Files.createDirectories(dumpDir);
  if (!Files.isWritable(dumpDir)) throw new AccessDeniedException(dumpDir.toString());
} catch (IOException e) {
  throw new RuntimeException("nfs.dump.dir not usable by gateway user: " + dumpDir, e);
}

When it happens

Trigger: RpcProgramNfs3 startup calls clearDirectory(config.get("nfs.dump.dir", "/tmp/.hdfs-nfs")): mkdirs() returns false when a plain file already exists at the path, a parent directory is not writable by the gateway user, the filesystem is read-only or full, or the earlier fullyDelete() of the existing dump dir left undeletable entries owned by another user.

Common situations: Running the NFS gateway as a user without write access to /tmp (hardened boxes with noexec/nodev or private /tmp per user); restarting the gateway after a crash that left files owned by root or a different user in the dump dir; dump dir placed on a read-only mount or a full disk; SELinux denying writes.

Related errors

• Permission denied: user=%s, path="%s":%s:%s:%s%s
• {} doesn't support modifyAclEntries
• {} doesn't support removeAclEntries
• {} doesn't support removeAcl
• {} doesn't support setAcl

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5e8b38a0459123f3. Report an issue: GitHub.