{"record":{"id":"5e8b38a0459123f3","repo":"apache/hadoop","slug":"cannot-create-dump-directory","errorCode":null,"errorMessage":"Cannot create dump directory {}","messagePattern":"Cannot create dump directory (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java","lineNumber":243,"sourceCode":"    String displayName = DNS.getDefaultHost(\"default\", \"default\")\n        + config.getInt(NfsConfigKeys.DFS_NFS_SERVER_PORT_KEY,\n            NfsConfigKeys.DFS_NFS_SERVER_PORT_DEFAULT);\n    metrics = Nfs3Metrics.create(config, displayName);\n    return new RpcProgramNfs3(config, registrationSocket, allowInsecurePorts);\n  }\n  \n  private void clearDirectory(String writeDumpDir) throws IOException {\n    File dumpDir = new File(writeDumpDir);\n    if (dumpDir.exists()) {\n      LOG.info(\"Delete current dump directory {}\", writeDumpDir);\n      if (!(FileUtil.fullyDelete(dumpDir))) {\n        throw new IOException(\"Cannot remove current dump directory: \"\n            + dumpDir);\n      }\n    }\n    LOG.info(\"Create new dump directory {}\", writeDumpDir);\n    if (!dumpDir.mkdirs()) {\n      throw new IOException(\"Cannot create dump directory \" + dumpDir);\n    }\n  }\n  \n  @Override\n  public void startDaemons() {\n    if (pauseMonitor == null) {\n      pauseMonitor = new JvmPauseMonitor();\n      pauseMonitor.init(config);\n      pauseMonitor.start();\n      metrics.getJvmMetrics().setPauseMonitor(pauseMonitor);\n    }\n    writeManager.startAsyncDataService();\n    try {\n      infoServer.start();\n    } catch (IOException e) {\n      LOG.error(\"failed to start web server\", e);\n    }\n  }","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java#L225-L261","documentation":"Thrown by the HDFS NFS3 gateway during initialization when it cannot create (or recreate) its write-dump directory, the local staging area where WRITE RPC payloads received before COMMIT are persisted (nfs.file.dump=true). clearDirectory() first fully deletes the existing dump dir and then calls mkdirs(); if mkdirs() returns false the gateway throws this IOException and refuses to start, because async WRITE handling depends on that directory (nfs.dump.dir, default /tmp/.hdfs-nfs).","triggerScenarios":"RpcProgramNfs3 startup calls clearDirectory(config.get(\"nfs.dump.dir\", \"/tmp/.hdfs-nfs\")): mkdirs() returns false when a plain file already exists at the path, a parent directory is not writable by the gateway user, the filesystem is read-only or full, or the earlier fullyDelete() of the existing dump dir left undeletable entries owned by another user.","commonSituations":"Running the NFS gateway as a user without write access to /tmp (hardened boxes with noexec/nodev or private /tmp per user); restarting the gateway after a crash that left files owned by root or a different user in the dump dir; dump dir placed on a read-only mount or a full disk; SELinux denying writes.","solutions":["Check the path and permissions: ls -ld <nfs.dump.dir value> (default /tmp/.hdfs-nfs) and confirm the gateway user can create/delete entries in its parent","Remove the stale dump directory left by a previous run as the right user: rm -rf /tmp/.hdfs-nfs (or chown it back to the gateway user with sudo)","Point nfs.dump.dir in core-site.xml at a dedicated, always-writable location with free disk space and restart the NFS3 gateway","Rule out a plain file with the same name, SELinux denials, and a full/read-only filesystem on that mount"],"exampleFix":"<!-- before: default shared tmp path the gateway user cannot (re)create -->\n<property>\n  <name>nfs.dump.dir</name>\n  <value>/tmp/.hdfs-nfs</value>\n</property>\n<!-- after: dedicated writable location, mkdir + chown nfs-user first -->\n<property>\n  <name>nfs.dump.dir</name>\n  <value>/var/nfs3/dump</value>\n</property>","handlingStrategy":"validation","validationCode":"// Pre-flight before starting the NFS3 gateway\nimport java.nio.file.*;\n\nPath dumpDir = Paths.get(conf.get(\"nfs.dump.dir\", \"/tmp/.hdfs-nfs\"));\ntry {\n  if (Files.exists(dumpDir)) {\n    try (var s = Files.list(dumpDir)) {\n      for (Path p : (Iterable<Path>) s::iterator) Files.deleteIfExists(p);\n    }\n    Files.deleteIfExists(dumpDir);\n  }\n  Files.createDirectories(dumpDir);\n  if (!Files.isWritable(dumpDir)) throw new AccessDeniedException(dumpDir.toString());\n} catch (IOException e) {\n  throw new RuntimeException(\"nfs.dump.dir not usable by gateway user: \" + dumpDir, e);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Give the NFS gateway a dedicated nfs.dump.dir owned by its run user instead of relying on /tmp","After crashes or user changes, clean the dump dir before restarting the gateway","Monitor free space and write access on the filesystem holding the dump dir","Prefer nfs.file.dump=false on read-mostly deployments where WRITE dump staging is not needed"],"tags":["hadoop","hdfs","nfs","nfs3-gateway","filesystem","permissions","startup"],"backgroundTag":"directory-creation-permission-denied","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}