{"record":{"id":"bd87190b6c5a6515","repo":"apache/hadoop","slug":"unable-to-delete-paxos-file-journal-id","errorCode":null,"errorMessage":"Unable to delete paxos file {} ; journal id: {}","messagePattern":"Unable to delete paxos file (.+?) ; journal id: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java","lineNumber":708,"sourceCode":"  public synchronized void purgeLogsOlderThan(RequestInfo reqInfo,\n      long minTxIdToKeep) throws IOException {\n    checkFormatted();\n    checkRequest(reqInfo);\n    \n    storage.purgeDataOlderThan(minTxIdToKeep);\n  }\n  \n  /**\n   * Remove the previously-recorded 'accepted recovery' information\n   * for a given log segment, once it is no longer necessary. \n   * @param segmentTxId the transaction ID to purge\n   * @throws IOException if the file could not be deleted\n   */\n  private void purgePaxosDecision(long segmentTxId) throws IOException {\n    File paxosFile = storage.getPaxosFile(segmentTxId);\n    if (paxosFile.exists()) {\n      if (!paxosFile.delete()) {\n        throw new IOException(\"Unable to delete paxos file \" + paxosFile +\n            \" ; journal id: \" + journalId);\n      }\n    }\n  }\n\n  /**\n   * @see QJournalProtocol#getEditLogManifest(String, String, long, boolean)\n   */\n  public RemoteEditLogManifest getEditLogManifest(long sinceTxId,\n      boolean inProgressOk) throws IOException {\n    // No need to checkRequest() here - anyone may ask for the list\n    // of segments.\n    checkFormatted();\n    \n    List<RemoteEditLog> logs = fjm.getRemoteEditLogs(sinceTxId, inProgressOk);\n    \n    if (inProgressOk) {\n      RemoteEditLog log = null;","sourceCodeStart":690,"sourceCodeEnd":726,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java#L690-L726","documentation":"Journal.purgePaxosDecision throws this IOException when, after a recovery decision has been applied, File.delete() returns false for the Paxos acceptance file (paxos file for a segment txid) on the JournalNode. The decision file records which recovery proposal was accepted; once the recovered segment is in place it is no longer needed. A false return from delete() means the OS refused removal while the file still exists — almost always permissions, an open file handle, or an NFS/consistency issue on the journal directory.","triggerScenarios":"Called at the end of journal recovery (Journal.resyncPhase2/commit) or when a segment is deleted after being superseded; paxosFile.exists() is true but paxosFile.delete() returns false. Common with the journal dir on NFS, a concurrently open file handle (another process scanning the dir), read-only mount, or a wrong owner after dir migration.","commonSituations":"Journal directories hosted on NFS or a network filer with silly-rename semantics; running the JournalNode as a different user than the one that owns the paxos files (e.g., after user migration or container switch); a leftover indexer/AVG scanner holding the file open; disk mounted read-only after an error.","solutions":["Check ownership/permissions of the paxos file under dfs.journalnode.edits.dir/<jid>/paxos-data and chown/chmod so the JournalNode user can delete (rwx on the parent directory is what matters).","Find and stop whatever holds the file open (lsof | grep paxos; on NFS, stale .nfsXXXX silly-rename files are the tell), then let the next recovery retry the purge.","Verify the journal filesystem is mounted read-write (mount, nfs remount) and has no disk errors (dmesg); remount or fix the filer export.","As a last resort on a quiescent journal, manually delete the leftover paxos file while the JournalNode is stopped — it is only a recovery-decision record."],"exampleFix":"# before: purge fails during recovery\n# IOException: Unable to delete paxos file /jndir/myjournal/paxos-data/1000-0.txid ; journal id: myjournal\n\n# after: give the JournalNode user control of the dir, clear holders, retry recovery\nsudo chown -R hdfs:hadoop /jndir/myjournal\nsudo chmod -R u+rwX /jndir/myjournal\nlsof +D /jndir/myjournal   # kill any stale holder, then retry the NN failover/recovery","handlingStrategy":"retry","validationCode":"File paxos = storage.getPaxosFile(segmentTxId);\nif (paxos.exists() && !paxos.getParentFile().canWrite()) {\n  throw new IOException(\"No write permission on paxos dir; fix before recovery\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  // recovery path that purges paxos decisions\n  journalResync();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Unable to delete paxos file\")) {\n    fixOwnershipAndOpenHandles(paxosDir); // then retry recovery once\n  } else { throw e; }\n}","preventionTips":["Keep journal directories on local POSIX filesystems, not NFS.","Ensure the JournalNode user owns the entire edits dir tree (rwx on directories).","Exclude journal dirs from AV/indexing/backup agents that hold files open during recovery."],"tags":["hdfs","qjournal","paxos","file-delete","permissions","nfs"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}