{"record":{"id":"cdc4ecf3be868154","repo":"apache/hadoop","slug":"journal-disabled-until-next-roll","errorCode":null,"errorMessage":"Journal disabled until next roll","messagePattern":"Journal disabled until next roll","errorType":"exception","errorClass":"JournalOutOfSyncException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java","lineNumber":421,"sourceCode":"            unreserveQueueSpace(data.length);\n          }\n\n          @Override\n          public void onSuccess(Void t) {\n            unreserveQueueSpace(data.length);\n          }\n        }, MoreExecutors.directExecutor());\n      }\n    }\n    return ret;\n  }\n\n  private void throwIfOutOfSync() throws IOException {\n    if (isOutOfSync()) {\n      // Even if we're out of sync, it's useful to send an RPC\n      // to the remote node in order to update its lag metrics, etc.\n      heartbeatIfNecessary();\n      throw new JournalOutOfSyncException(\"Journal disabled until next roll\");\n    }\n  }\n\n  /**\n   * When we've entered an out-of-sync state, it's still useful to periodically\n   * send an empty RPC to the server, such that it has the up to date\n   * committedTxId. This acts as a sanity check during recovery, and also allows\n   * that node's metrics to be up-to-date about its lag.\n   * \n   * In the future, this method may also be used in order to check that the\n   * current node is still the current writer, even if no edits are being\n   * written.\n   */\n  private void heartbeatIfNecessary() throws IOException {\n    if (lastHeartbeatStopwatch.now(TimeUnit.MILLISECONDS)\n        > HEARTBEAT_INTERVAL_MILLIS || !lastHeartbeatStopwatch.isRunning()) {\n      try {\n        getProxy().heartbeat(createReqInfo());","sourceCodeStart":403,"sourceCodeEnd":439,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java#L403-L439","documentation":"IPCLoggerChannel tracks whether its JournalNode is known to be in sync with the quorum's edit history. After failures that leave doubt (its acks lagged while the quorum committed), the channel is marked out-of-sync; throwIfOutOfSync() first sends a heartbeat so the JN can update its lag metrics, then refuses writes with JournalOutOfSyncException. Writes to that JN stay disabled until the segment is rolled, which re-establishes a known-good starting point.","triggerScenarios":"Any write-path call on a logger channel previously marked out-of-sync — e.g. that JN timed out during journalEdits while the majority committed, so subsequent writes through the channel throw. The channel rejoins automatically at the next startLogSegment/finalize (segment roll).","commonSituations":"Transient JournalNode outage in a 3-node quorum; network flap between NN and one JN; commonly seen in logs during or after recovery. AsyncLoggerSet tolerates it as long as a majority stays healthy, so it degrades redundancy rather than stopping edits.","solutions":["Treat it as per-JournalNode degradation: find why that JN fell behind (process, disk, network) and fix it.","Wait for or trigger a log roll — hdfs dfsadmin -rollEdits or the automatic roll at segment close — after which the JN rejoins the quorum.","If the channel never resyncs, failover or restart the active NameNode so QJM recovery (new epoch plus segment recovery) re-establishes sync.","Monitor JournalNode lag metrics instead of alerting on the single exception."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  channel.sendEdits(...);\n} catch (JournalOutOfSyncException e) {\n  // expected after this logger missed commits: quorum continues without it;\n  // it rejoins at the next segment roll — optionally trigger one\n  LOG.warn(\"JournalNode out of sync until next roll: {}\", channel, e);\n  maybeRollEdits(); // e.g. hdfs dfsadmin -rollEdits\n}","preventionTips":["Do not alert on a single JournalOutOfSyncException; alert when it persists across rolls or a second JN degrades.","Monitor JournalNode lag metrics to catch the slow JN before it goes out of sync.","Trigger or await a log roll (segment finalize/start) after fixing a lagging JN so it rejoins promptly."],"tags":["hdfs","qjm","journal-node","out-of-sync","edit-log","ha"],"backgroundTag":"quorum-node-out-of-sync","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}