apache/hadoop · error · IOException

Unable to delete paxos file {} ; journal id: {}

Error message

Unable to delete paxos file {} ; journal id: {}

What it means

Journal.purgePaxosDecision throws this IOException when, after a recovery decision has been applied, File.delete() returns false for the Paxos acceptance file (paxos file for a segment txid) on the JournalNode. The decision file records which recovery proposal was accepted; once the recovered segment is in place it is no longer needed. A false return from delete() means the OS refused removal while the file still exists — almost always permissions, an open file handle, or an NFS/consistency issue on the journal directory.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:708

  public synchronized void purgeLogsOlderThan(RequestInfo reqInfo,
      long minTxIdToKeep) throws IOException {
    checkFormatted();
    checkRequest(reqInfo);
    
    storage.purgeDataOlderThan(minTxIdToKeep);
  }
  
  /**
   * Remove the previously-recorded 'accepted recovery' information
   * for a given log segment, once it is no longer necessary. 
   * @param segmentTxId the transaction ID to purge
   * @throws IOException if the file could not be deleted
   */
  private void purgePaxosDecision(long segmentTxId) throws IOException {
    File paxosFile = storage.getPaxosFile(segmentTxId);
    if (paxosFile.exists()) {
      if (!paxosFile.delete()) {
        throw new IOException("Unable to delete paxos file " + paxosFile +
            " ; journal id: " + journalId);
      }
    }
  }

  /**
   * @see QJournalProtocol#getEditLogManifest(String, String, long, boolean)
   */
  public RemoteEditLogManifest getEditLogManifest(long sinceTxId,
      boolean inProgressOk) throws IOException {
    // No need to checkRequest() here - anyone may ask for the list
    // of segments.
    checkFormatted();
    
    List<RemoteEditLog> logs = fjm.getRemoteEditLogs(sinceTxId, inProgressOk);
    
    if (inProgressOk) {
      RemoteEditLog log = null;

View on GitHub (pinned to 2add963021)

Solutions

1. Check ownership/permissions of the paxos file under dfs.journalnode.edits.dir/<jid>/paxos-data and chown/chmod so the JournalNode user can delete (rwx on the parent directory is what matters).
2. Find and stop whatever holds the file open (lsof | grep paxos; on NFS, stale .nfsXXXX silly-rename files are the tell), then let the next recovery retry the purge.
3. Verify the journal filesystem is mounted read-write (mount, nfs remount) and has no disk errors (dmesg); remount or fix the filer export.
4. As a last resort on a quiescent journal, manually delete the leftover paxos file while the JournalNode is stopped — it is only a recovery-decision record.

Example fix

# before: purge fails during recovery
# IOException: Unable to delete paxos file /jndir/myjournal/paxos-data/1000-0.txid ; journal id: myjournal

# after: give the JournalNode user control of the dir, clear holders, retry recovery
sudo chown -R hdfs:hadoop /jndir/myjournal
sudo chmod -R u+rwX /jndir/myjournal
lsof +D /jndir/myjournal   # kill any stale holder, then retry the NN failover/recovery

File paxos = storage.getPaxosFile(segmentTxId);
if (paxos.exists() && !paxos.getParentFile().canWrite()) {
  throw new IOException("No write permission on paxos dir; fix before recovery");
}

try {
  // recovery path that purges paxos decisions
  journalResync();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to delete paxos file")) {
    fixOwnershipAndOpenHandles(paxosDir); // then retry recovery once
  } else { throw e; }
}

When it happens

Trigger: Called at the end of journal recovery (Journal.resyncPhase2/commit) or when a segment is deleted after being superseded; paxosFile.exists() is true but paxosFile.delete() returns false. Common with the journal dir on NFS, a concurrently open file handle (another process scanning the dir), read-only mount, or a wrong owner after dir migration.

Common situations: Journal directories hosted on NFS or a network filer with silly-rename semantics; running the JournalNode as a different user than the one that owns the paxos files (e.g., after user migration or container switch); a leftover indexer/AVG scanner holding the file open; disk mounted read-only after an error.

Related errors

• Cannot create dump directory {}
• Permission denied: user=%s, path="%s":%s:%s:%s%s
• {} doesn't support modifyAclEntries
• {} doesn't support removeAclEntries
• {} doesn't support removeAcl

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bd87190b6c5a6515. Report an issue: GitHub.