apache/hadoop · warning · JournalOutOfSyncException

Journal disabled until next roll

Error message

Journal disabled until next roll

What it means

IPCLoggerChannel tracks whether its JournalNode is known to be in sync with the quorum's edit history. After failures that leave doubt (its acks lagged while the quorum committed), the channel is marked out-of-sync; throwIfOutOfSync() first sends a heartbeat so the JN can update its lag metrics, then refuses writes with JournalOutOfSyncException. Writes to that JN stay disabled until the segment is rolled, which re-establishes a known-good starting point.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:421

            unreserveQueueSpace(data.length);
          }

          @Override
          public void onSuccess(Void t) {
            unreserveQueueSpace(data.length);
          }
        }, MoreExecutors.directExecutor());
      }
    }
    return ret;
  }

  private void throwIfOutOfSync() throws IOException {
    if (isOutOfSync()) {
      // Even if we're out of sync, it's useful to send an RPC
      // to the remote node in order to update its lag metrics, etc.
      heartbeatIfNecessary();
      throw new JournalOutOfSyncException("Journal disabled until next roll");
    }
  }

  /**
   * When we've entered an out-of-sync state, it's still useful to periodically
   * send an empty RPC to the server, such that it has the up to date
   * committedTxId. This acts as a sanity check during recovery, and also allows
   * that node's metrics to be up-to-date about its lag.
   * 
   * In the future, this method may also be used in order to check that the
   * current node is still the current writer, even if no edits are being
   * written.
   */
  private void heartbeatIfNecessary() throws IOException {
    if (lastHeartbeatStopwatch.now(TimeUnit.MILLISECONDS)
        > HEARTBEAT_INTERVAL_MILLIS || !lastHeartbeatStopwatch.isRunning()) {
      try {
        getProxy().heartbeat(createReqInfo());

View on GitHub (pinned to 2add963021)

Solutions

  1. Treat it as per-JournalNode degradation: find why that JN fell behind (process, disk, network) and fix it.
  2. Wait for or trigger a log roll — hdfs dfsadmin -rollEdits or the automatic roll at segment close — after which the JN rejoins the quorum.
  3. If the channel never resyncs, failover or restart the active NameNode so QJM recovery (new epoch plus segment recovery) re-establishes sync.
  4. Monitor JournalNode lag metrics instead of alerting on the single exception.
Defensive patterns

Strategy: fallback

Try / catch

try {
  channel.sendEdits(...);
} catch (JournalOutOfSyncException e) {
  // expected after this logger missed commits: quorum continues without it;
  // it rejoins at the next segment roll — optionally trigger one
  LOG.warn("JournalNode out of sync until next roll: {}", channel, e);
  maybeRollEdits(); // e.g. hdfs dfsadmin -rollEdits
}

Prevention

When it happens

Trigger: Any write-path call on a logger channel previously marked out-of-sync — e.g. that JN timed out during journalEdits while the majority committed, so subsequent writes through the channel throw. The channel rejoins automatically at the next startLogSegment/finalize (segment roll).

Common situations: Transient JournalNode outage in a 3-node quorum; network flap between NN and one JN; commonly seen in logs during or after recovery. AsyncLoggerSet tolerates it as long as a majority stays healthy, so it degrades redundancy rather than stopping edits.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cdc4ecf3be868154. Report an issue: GitHub.