apache/hadoop · critical · IOException

failed for too many journals

Error message

 failed for too many journals

What it means

JournalSet disables every journal (QJM, local edits dir, NFS) that failed an edit-log operation, then asks NameNodeResourcePolicy.areResourcesAvailable whether at least minimumRedundantJournals (dfs.namenode.edits.dir.minimum, default 1) healthy journals remain. When the surviving set falls below that minimum this IOException is thrown, and on the NameNode it normally terminates the process because edit-log durability can no longer be guaranteed. The per-journal errors immediately above this line in the log name the failing journals.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java:420

          abortAllJournals();
          // the current policy is to shutdown the NN on errors to shared edits
          // dir. There are many code paths to shared edits failures - syncs,
          // roll of edits etc. All of them go through this common function 
          // where the isRequired() check is made. Applying exit policy here 
          // to catch all code paths.
          terminate(1, msg);
        } else {
          LOG.error("Error: " + status + " failed for (journal " + jas + ")", t);
          badJAS.add(jas);          
        }
      }
    }
    disableAndReportErrorOnJournals(badJAS);
    if (!NameNodeResourcePolicy.areResourcesAvailable(journals,
        minimumRedundantJournals)) {
      String message = status + " failed for too many journals";
      LOG.error("Error: " + message);
      throw new IOException(message);
    }
  }
  
  /**
   * Abort all of the underlying streams.
   */
  private void abortAllJournals() {
    for (JournalAndStream jas : journals) {
      if (jas.isActive()) {
        jas.abort();
      }
    }
  }

  /**
   * An implementation of EditLogOutputStream that applies a requested method on
   * all the journals that are currently active.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Read the directly preceding 'Error: <status> failed for (journal <jas>)' lines - they name each failing journal and its cause; fix those first.
2. Restore at least dfs.namenode.edits.dir.minimum healthy journals: remount NFS, fix permissions/space on local dirs, bring enough JournalNodes back for QJM quorum.
3. Re-check dfs.namenode.edits.dir for duplicated or mistyped paths that silently reduce real redundancy.
4. Only if the durability policy allows, lower dfs.namenode.edits.dir.minimum - not recommended, since it exists to stop silent single-journal writes.

Example fix

// before - single point of failure for the edit log
<property>
  <name>dfs.namenode.edits.dir</name>
  <value>file:///data/dfs/edits</value>
</property>

// after - two independent journal media, redundancy survives one failure
<property>
  <name>dfs.namenode.edits.dir</name>
  <value>qjm://jn1:8485;jn2:8485;jn3:8485/mycluster, file:///data/dfs/edits</value>
</property>

// Config lint: redundancy must survive the loss of any single journal
int minimum = conf.getInt("dfs.namenode.edits.dir.minimum", 1);
Set<String> distinctDirs = new HashSet<>(Arrays.asList(
    conf.getTrimmedStrings("dfs.namenode.edits.dir")));
if (distinctDirs.size() <= minimum) {
  LOG.warn("Edit-log redundancy at risk: {} journals configured, minimum {} - one failure stops the NN",
      distinctDirs.size(), minimum);
}

try {
  editLog.logEdit(...);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("failed for too many journals")) {
    // NN cannot continue safely: page on-call, restore journals, restart from recovered state
    LOG.fatal("Edit-log redundancy exhausted - NameNode must stop", e);
    Runtime.getRuntime().halt(1);
  }
  throw e;
}

When it happens

Trigger: An edit-log operation (logEdit, rollEditLog, purgeLogsOlderThan) fails on several journals at once - QuorumJournalManager loses quorum while the local/NFS edits dir also errors (permission, disk full, stale NFS) - leaving fewer than dfs.namenode.edits.dir.minimum writable journals.

Common situations: Only one journal configured and it fails; NFS edits mount down while QJM is simultaneously degraded; duplicate/typo'd dfs.namenode.edits.dir entries counting as one; minimum raised above the number of healthy journals after removing a JournalNode.

Related errors

• No edits file for txid " + startTxId + "-" + endTxId + " exi
• Cannot remove current directory: {curDir}
• Cannot create directory {curDir}
• Can't format the storage directory because the current direc
• Cannot create directory {rootPath}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/44ce126023ef53bf. Report an issue: GitHub.