{"record":{"id":"e61d280b7d414c21","repo":"apache/hadoop","slug":"cannot-remove-current-dump-directory","errorCode":null,"errorMessage":"Cannot remove current dump directory: {}","messagePattern":"Cannot remove current dump directory: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java","lineNumber":237,"sourceCode":"  }\n\n  public static RpcProgramNfs3 createRpcProgramNfs3(NfsConfiguration config,\n      DatagramSocket registrationSocket, boolean allowInsecurePorts)\n      throws IOException {\n    DefaultMetricsSystem.initialize(\"Nfs3\");\n    String displayName = DNS.getDefaultHost(\"default\", \"default\")\n        + config.getInt(NfsConfigKeys.DFS_NFS_SERVER_PORT_KEY,\n            NfsConfigKeys.DFS_NFS_SERVER_PORT_DEFAULT);\n    metrics = Nfs3Metrics.create(config, displayName);\n    return new RpcProgramNfs3(config, registrationSocket, allowInsecurePorts);\n  }\n  \n  private void clearDirectory(String writeDumpDir) throws IOException {\n    File dumpDir = new File(writeDumpDir);\n    if (dumpDir.exists()) {\n      LOG.info(\"Delete current dump directory {}\", writeDumpDir);\n      if (!(FileUtil.fullyDelete(dumpDir))) {\n        throw new IOException(\"Cannot remove current dump directory: \"\n            + dumpDir);\n      }\n    }\n    LOG.info(\"Create new dump directory {}\", writeDumpDir);\n    if (!dumpDir.mkdirs()) {\n      throw new IOException(\"Cannot create dump directory \" + dumpDir);\n    }\n  }\n  \n  @Override\n  public void startDaemons() {\n    if (pauseMonitor == null) {\n      pauseMonitor = new JvmPauseMonitor();\n      pauseMonitor.init(config);\n      pauseMonitor.start();\n      metrics.getJvmMetrics().setPauseMonitor(pauseMonitor);\n    }\n    writeManager.startAsyncDataService();","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java#L219-L255","documentation":"On startup RpcProgramNfs3.clearDirectory() wipes the NFS write dump directory (nfs.dump.dir, default /tmp/.hdfs-nfs) to start from a clean state. If FileUtil.fullyDelete() cannot remove the existing directory, startup fails with this IOException (a companion check then fails on mkdirs if creation is also impossible).","triggerScenarios":"The dump directory contains files the gateway user cannot delete: leftovers owned by root or a previous different service account, a read-only mount, SELinux denials, or the path pointing at a file/symlink rather than a directory.","commonSituations":"Switching the user the NFS gateway runs as (old dump files keep the former owner); /tmp cleaner daemons partially deleting the directory leaving odd ownership; restoring gateway state from images/backups that preserved stale ownership.","solutions":["Manually remove the dump directory as a privileged user: rm -rf /tmp/.hdfs-nfs (or your nfs.dump.dir), then restart the gateway.","Configure nfs.dump.dir to a dedicated directory owned by the gateway user instead of /tmp.","Always run the NFS gateway under the same service account so dump files stay deletable."],"exampleFix":"# before: gateway startup fails with 'Cannot remove current dump directory'\nsudo rm -rf /tmp/.hdfs-nfs\n# after (nfssite-site.xml + ownership fix)\n<property><name>nfs.dump.dir</name><value>/var/lib/hadoop-nfs/dump</value></property>\nchown -R hdfs:hdfs /var/lib/hadoop-nfs","handlingStrategy":"validation","validationCode":"/* pre-start check: dump dir must be removable by the gateway user */\nFile dumpDir = new File(conf.get(\"nfs.dump.dir\", \"/tmp/.hdfs-nfs\"));\nif (dumpDir.exists() && !Files.isWritable(dumpDir.getParentFile())) {\n    throw new IOException(\"Cannot manage dump dir \" + dumpDir + \" — fix ownership\");\n}\nfor (File f : dumpDir.listFiles() != null ? dumpDir.listFiles() : new File[0]) {\n    if (!Files.isWritable(f.toPath().getParent()) && !f.canWrite()) {\n        throw new IOException(\"Undeletable leftover in dump dir: \" + f);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    /* gateway start */\n} catch (IOException e) {\n    if (e.getMessage().contains(\"Cannot remove current dump directory\")) {\n        // deterministic: clear the dir as its owner (or privileged), e.g.\n        //   sudo rm -rf /tmp/.hdfs-nfs\n        // then restart. Do not blindly delete if other services share the path.\n    } else {\n        throw e;\n    }\n}","preventionTips":["Give the NFS gateway a dedicated nfs.dump.dir (not /tmp) owned by its service account.","Always run the gateway as the same user so leftover dump files remain deletable.","Include 'rm -rf <nfs.dump.dir>/*' (as the right user) in gateway restart runbooks.","After crashes, inspect ownership of dump-dir leftovers before restarting."],"tags":["nfs-gateway","hdfs","dump-directory","permissions","startup","filesystem"],"backgroundTag":"directory-cleanup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}