{"record":{"id":"1bb4b47ab9fb12f6","repo":"apache/hadoop","slug":"results-differed-for-canrollback","errorCode":null,"errorMessage":"Results differed for canRollBack","messagePattern":"Results differed for canRollBack","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":738,"sourceCode":"  public boolean canRollBack(StorageInfo storage, StorageInfo prevStorage,\n      int targetLayoutVersion) throws IOException {\n    QuorumCall<AsyncLogger, Boolean> call = loggers.canRollBack(storage,\n        prevStorage, targetLayoutVersion);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"lockSharedStorage\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not check if roll back possible for\"\n            + \" one or more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can\n      // just return the first result.\n      try {\n        DFSUtil.assertAllResultsEqual(call.getResults().values());\n      } catch (AssertionError ae) {\n        throw new IOException(\"Results differed for canRollBack\", ae);\n      }\n      for (Boolean result : call.getResults().values()) {\n        return result;\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for lockSharedStorage() \" +\n          \"response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for lockSharedStorage() \" +\n          \"response\");\n    }\n    \n    throw new AssertionError(\"Unreachable code.\");\n  }\n\n  @Override\n  public void doRollback() throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doRollback();","sourceCodeStart":720,"sourceCodeEnd":756,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L720-L756","documentation":"QJM canRollBack() asks every JournalNode whether its on-disk storage can roll back to the previous layout and then requires all answers to be identical (DFSUtil.assertAllResultsEqual). If any JN returns a different boolean, the AssertionError is wrapped in this IOException. It signals that the JournalNodes' storage states have diverged from each other.","triggerScenarios":"Calling canRollBack() (during NameNode rollback checks, e.g. 'hdfs namenode -rollback' preparation) when one JN's directory is on a different layoutVersion/cTime state than the others — for example one JN was already rolled back or upgraded while the rest were not, or a JN directory was restored from an older backup.","commonSituations":"Retrying a rollback after a previous attempt failed halfway; restoring a single JN's journal directory from an old snapshot; deleting or hand-editing one JN's current dir; a JN that was down during a cluster upgrade and rejoined later.","solutions":["Compare the VERSION file (layoutVersion, cTime, namespaceID) in every JN's current dir under dfs.journalnode.edits.dir/<journal-id>/current/ across all JournalNodes to find the odd one out.","If most JNs agree, bring the divergent JN back in sync: restore its journal dir from a healthy JN's copy, or drop its data so shared edits can be rebuilt.","After reconciling NN-side storage, run 'hdfs namenode -initializeSharedEdits' to rebuild consistent shared edits across all JNs, then retry the rollback.","If the divergence is unclear, run with the rollback tooling on a copy of the dirs in a scratch environment before mutating production JN state."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Preflight: compare the answer each JN would give before rolling back\nMap<AsyncLogger, Boolean> perJn = collectCanRollBackPerLogger();\nif (new HashSet<>(perJn.values()).size() > 1) {\n  throw new IllegalStateException(\"JNs disagree on rollback: \" + perJn);\n}","typeGuard":"static boolean isResultMismatch(IOException ioe) {\n  return ioe.getMessage() != null && ioe.getMessage().startsWith(\"Results differed\");\n}","tryCatchPattern":"try {\n  qjm.canRollBack(storage, prevStorage, targetLayoutVersion);\n} catch (IOException ioe) {\n  if (isResultMismatch(ioe)) {\n    // NOT retryable: reconcile JN storage dirs first, then re-run\n    haltAndAlertAdmin(ioe);\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Always keep JN journal directories in lockstep: never restore just one JN from an arbitrary backup.","Script a VERSION-file diff across all JNs as a pre-upgrade/pre-rollback gate.","Never edit or delete files inside a JN's current dir by hand."],"tags":["qjm","rollback","state-divergence","consensus-check"],"backgroundTag":"quorum-result-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}