{"record":{"id":"822e5e8e4bbfc488","repo":"apache/hadoop","slug":"timed-out-waiting-for-locksharedstorage-response","errorCode":null,"errorMessage":"Timed out waiting for lockSharedStorage() response","messagePattern":"Timed out waiting for lockSharedStorage\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":747,"sourceCode":"        call.rethrowException(\"Could not check if roll back possible for\"\n            + \" one or more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can\n      // just return the first result.\n      try {\n        DFSUtil.assertAllResultsEqual(call.getResults().values());\n      } catch (AssertionError ae) {\n        throw new IOException(\"Results differed for canRollBack\", ae);\n      }\n      for (Boolean result : call.getResults().values()) {\n        return result;\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for lockSharedStorage() \" +\n          \"response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for lockSharedStorage() \" +\n          \"response\");\n    }\n    \n    throw new AssertionError(\"Unreachable code.\");\n  }\n\n  @Override\n  public void doRollback() throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doRollback();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doRollback\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not perform rollback of one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doFinalize() response\");","sourceCodeStart":729,"sourceCodeEnd":765,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L729-L765","documentation":"The canRollBack() quorum call did not receive responses from all JournalNodes within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms) and waitFor threw TimeoutException. The message text says 'lockSharedStorage()' — a copy-paste quirk; the actual operation is canRollBack. Unlike normal QJM writes, this wait demands an answer from every JN, not a quorum.","triggerScenarios":"Running the rollback check while one or more JournalNodes are down, unreachable, or slow (disk/GC/network); any single JN that fails to answer within dfs.qjm.operations.timeout produces this timeout.","commonSituations":"A JN host is down when an operator starts rollback; a JN disk is dying so reading storage metadata takes minutes; congested network between NN and a JN; default 60s timeout too tight for large/slow journal dirs.","solutions":["Bring all JournalNodes up and reachable, then re-run the rollback command.","Check the unreachable JN's logs for disk or fsync slowness and remediate (replace disk, free space).","Increase dfs.qjm.operations.timeout if JNs are healthy but slow, and retry.","Restore a lost JN from a copy of another JN's journal directory before retrying — the call needs every JN to answer."],"exampleFix":"// hdfs-site.xml — give canRollBack room on slow JournalNodes\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>180000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Same preflight as other all-JN waits: all JournalNodes reachable\nboolean allJnUp = sharedEditsUris.stream()\n    .allMatch(u -> jnStatusAnswers(u));\nif (!allJnUp) {\n  throw new IllegalStateException(\"Refusing rollback check: some JNs down\");\n}","typeGuard":"static boolean isCanRollBackTimeout(IOException ioe) {\n  return ioe.getCause() instanceof TimeoutException\n      && ioe.getMessage().contains(\"Timed out waiting\");\n}","tryCatchPattern":"try {\n  qjm.canRollBack(storage, prevStorage, targetLayoutVersion);\n} catch (IOException ioe) {\n  if (ioe.getCause() instanceof TimeoutException) {\n    retryWithBackoff(); // after verifying JN health\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Run rollback only when all 3 (all configured) JournalNodes are confirmed up.","Alert on JN RPC latency so slow JNs are caught before procedures depend on them.","Remember the misleading message: 'lockSharedStorage()' here is really canRollBack timing out."],"tags":["qjm","rollback","timeout","journalnode","misleading-message"],"backgroundTag":"rpc-quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}