apache/hadoop · error · IOException

Results differed for canRollBack

Error message

Results differed for canRollBack

What it means

QJM canRollBack() asks every JournalNode whether its on-disk storage can roll back to the previous layout and then requires all answers to be identical (DFSUtil.assertAllResultsEqual). If any JN returns a different boolean, the AssertionError is wrapped in this IOException. It signals that the JournalNodes' storage states have diverged from each other.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:738

  public boolean canRollBack(StorageInfo storage, StorageInfo prevStorage,
      int targetLayoutVersion) throws IOException {
    QuorumCall<AsyncLogger, Boolean> call = loggers.canRollBack(storage,
        prevStorage, targetLayoutVersion);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "lockSharedStorage");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not check if roll back possible for"
            + " one or more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can
      // just return the first result.
      try {
        DFSUtil.assertAllResultsEqual(call.getResults().values());
      } catch (AssertionError ae) {
        throw new IOException("Results differed for canRollBack", ae);
      }
      for (Boolean result : call.getResults().values()) {
        return result;
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for lockSharedStorage() " +
          "response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for lockSharedStorage() " +
          "response");
    }
    
    throw new AssertionError("Unreachable code.");
  }

  @Override
  public void doRollback() throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doRollback();

View on GitHub (pinned to 2add963021)

Solutions

1. Compare the VERSION file (layoutVersion, cTime, namespaceID) in every JN's current dir under dfs.journalnode.edits.dir/<journal-id>/current/ across all JournalNodes to find the odd one out.
2. If most JNs agree, bring the divergent JN back in sync: restore its journal dir from a healthy JN's copy, or drop its data so shared edits can be rebuilt.
3. After reconciling NN-side storage, run 'hdfs namenode -initializeSharedEdits' to rebuild consistent shared edits across all JNs, then retry the rollback.
4. If the divergence is unclear, run with the rollback tooling on a copy of the dirs in a scratch environment before mutating production JN state.

// Preflight: compare the answer each JN would give before rolling back
Map<AsyncLogger, Boolean> perJn = collectCanRollBackPerLogger();
if (new HashSet<>(perJn.values()).size() > 1) {
  throw new IllegalStateException("JNs disagree on rollback: " + perJn);
}

static boolean isResultMismatch(IOException ioe) {
  return ioe.getMessage() != null && ioe.getMessage().startsWith("Results differed");
}

try {
  qjm.canRollBack(storage, prevStorage, targetLayoutVersion);
} catch (IOException ioe) {
  if (isResultMismatch(ioe)) {
    // NOT retryable: reconcile JN storage dirs first, then re-run
    haltAndAlertAdmin(ioe);
  } else {
    throw ioe;
  }
}

When it happens

Trigger: Calling canRollBack() (during NameNode rollback checks, e.g. 'hdfs namenode -rollback' preparation) when one JN's directory is on a different layoutVersion/cTime state than the others — for example one JN was already rolled back or upgraded while the rest were not, or a JN directory was restored from an older backup.

Common situations: Retrying a rollback after a previous attempt failed halfway; restoring a single JN's journal directory from an old snapshot; deleting or hand-editing one JN's current dir; a JN that was down during a cluster upgrade and rejoined later.

Related errors

• Results differed for getJournalCTime
• Interrupted waiting for lockSharedStorage() response
• Timed out waiting for lockSharedStorage() response
• Interrupted waiting for discardSegments() response
• Timed out waiting for discardSegments() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1bb4b47ab9fb12f6. Report an issue: GitHub.