apache/hadoop · error · IOException

Results differed for getJournalCTime

Error message

Results differed for getJournalCTime

What it means

getJournalCTime() asks every JournalNode for its journal creation/upgrade time and requires all results to be equal (DFSUtil.assertAllResultsEqual). Divergent cTime values across JNs — meaning the nodes were formatted or upgraded at different moments — produce this IOException. The NN uses this during upgrade startup to sanity-check shared edits consistency.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:807

  
  @Override
  public long getJournalCTime() throws IOException {
    QuorumCall<AsyncLogger, Long> call = loggers.getJournalCTime();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0,
          timeoutMs, "getJournalCTime");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not journal CTime for one "
            + "more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can
      // just return the first result.
      try {
        DFSUtil.assertAllResultsEqual(call.getResults().values());
      } catch (AssertionError ae) {
        throw new IOException("Results differed for getJournalCTime", ae);
      }
      for (Long result : call.getResults().values()) {
        return result;
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for getJournalCTime() " +
          "response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for getJournalCTime() " +
          "response");
    }
    
    throw new AssertionError("Unreachable code.");
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Compare the cTime field in each JN's current dir VERSION file (dfs.journalnode.edits.dir/<jid>/current/VERSION) to identify the divergent node.
2. If the majority share one cTime, overwrite the odd JN's journal dir with a copy of a healthy JN's dir (stop the JN first, copy, restart).
3. Alternatively rebuild shared edits wholesale: stop NNs, correct NN storage, run 'hdfs namenode -initializeSharedEdits' to lay down identical state on all JNs.
4. Verify JN address configuration — a JN from another cluster in dfs.namenode.shared.edits.dir also shows up as a cTime mismatch.

// Preflight: fetch cTime from each JN and require unanimity
List<Long> ctimes = queryEachJnCtime(sharedEditsUris);
if (ctimes.stream().distinct().count() > 1) {
  throw new IllegalStateException("JN cTimes diverge: " + ctimes);
}

static boolean isCTimeMismatch(IOException ioe) {
  return ioe.getMessage() != null
      && ioe.getMessage().contains("Results differed for getJournalCTime");
}

try {
  long ct = qjm.getJournalCTime();
} catch (IOException ioe) {
  if (isCTimeMismatch(ioe)) {
    // data problem, not transient: stop the upgrade and reconcile JN dirs
    haltAndAlertAdmin(ioe);
  } else {
    throw ioe;
  }
}

When it happens

Trigger: Calling getJournalCTime() during upgrade-related startup when JN dirs were created/upgraded at different times: one JN initialized later, one JN restored from backup, one JN that missed a completed upgrade, or a JN reformatted by hand.

Common situations: Adding a replacement JournalNode that was freshly initialized while others kept original state; restoring one JN from an old backup; a JN that was down during an 'hdfs namenode -upgrade' and never caught up; mixing dirs from a different cluster.

Related errors

• Results differed for canRollBack
• Interrupted waiting for getJournalCTime() response
• Timed out waiting for getJournalCTime() response
• Interrupted waiting for doPreUpgrade() response
• Timed out waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5933d0bfa4bfc8a7. Report an issue: GitHub.