apache/hadoop · error · IOException

Timed out waiting for lockSharedStorage() response

Error message

Timed out waiting for lockSharedStorage() response

What it means

The canRollBack() quorum call did not receive responses from all JournalNodes within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms) and waitFor threw TimeoutException. The message text says 'lockSharedStorage()' — a copy-paste quirk; the actual operation is canRollBack. Unlike normal QJM writes, this wait demands an answer from every JN, not a quorum.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:747

        call.rethrowException("Could not check if roll back possible for"
            + " one or more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can
      // just return the first result.
      try {
        DFSUtil.assertAllResultsEqual(call.getResults().values());
      } catch (AssertionError ae) {
        throw new IOException("Results differed for canRollBack", ae);
      }
      for (Boolean result : call.getResults().values()) {
        return result;
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for lockSharedStorage() " +
          "response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for lockSharedStorage() " +
          "response");
    }
    
    throw new AssertionError("Unreachable code.");
  }

  @Override
  public void doRollback() throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doRollback();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doRollback");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not perform rollback of one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doFinalize() response");

View on GitHub (pinned to 2add963021)

Solutions

1. Bring all JournalNodes up and reachable, then re-run the rollback command.
2. Check the unreachable JN's logs for disk or fsync slowness and remediate (replace disk, free space).
3. Increase dfs.qjm.operations.timeout if JNs are healthy but slow, and retry.
4. Restore a lost JN from a copy of another JN's journal directory before retrying — the call needs every JN to answer.

Example fix

// hdfs-site.xml — give canRollBack room on slow JournalNodes
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>180000</value>
</property>

// Same preflight as other all-JN waits: all JournalNodes reachable
boolean allJnUp = sharedEditsUris.stream()
    .allMatch(u -> jnStatusAnswers(u));
if (!allJnUp) {
  throw new IllegalStateException("Refusing rollback check: some JNs down");
}

static boolean isCanRollBackTimeout(IOException ioe) {
  return ioe.getCause() instanceof TimeoutException
      && ioe.getMessage().contains("Timed out waiting");
}

try {
  qjm.canRollBack(storage, prevStorage, targetLayoutVersion);
} catch (IOException ioe) {
  if (ioe.getCause() instanceof TimeoutException) {
    retryWithBackoff(); // after verifying JN health
  } else {
    throw ioe;
  }
}

When it happens

Trigger: Running the rollback check while one or more JournalNodes are down, unreachable, or slow (disk/GC/network); any single JN that fails to answer within dfs.qjm.operations.timeout produces this timeout.

Common situations: A JN host is down when an operator starts rollback; a JN disk is dying so reading storage metadata takes minutes; congested network between NN and a JN; default 60s timeout too tight for large/slow journal dirs.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for discardSegments() response
• Timed out waiting for doFinalize() response
• Interrupted waiting for lockSharedStorage() response
• Timed out waiting for getJournalCTime() response
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/822e5e8e4bbfc488. Report an issue: GitHub.