apache/hadoop · error · IOException

Timed out waiting for getJournalCTime() response

Error message

Timed out waiting for getJournalCTime() response

What it means

getJournalCTime() waited for every JournalNode to report its cTime and the full set of responses did not arrive within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms); the TimeoutException is wrapped in this IOException. As elsewhere in these all-JN waits, one missing answer times out the call.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:816

        call.rethrowException("Could not journal CTime for one "
            + "more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can
      // just return the first result.
      try {
        DFSUtil.assertAllResultsEqual(call.getResults().values());
      } catch (AssertionError ae) {
        throw new IOException("Results differed for getJournalCTime", ae);
      }
      for (Long result : call.getResults().values()) {
        return result;
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for getJournalCTime() " +
          "response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for getJournalCTime() " +
          "response");
    }
    
    throw new AssertionError("Unreachable code.");
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Start all JournalNodes and confirm their RPC ports answer from the NN host, then retry the NN startup/upgrade.
2. Investigate the non-answering JN's logs for disk or GC stalls and fix them.
3. Increase dfs.qjm.operations.timeout if the JNs are healthy but occasionally slow.
4. Restore a permanently lost JN from another JN's journal directory before retrying.

Example fix

// hdfs-site.xml
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>120000</value>
</property>

// Confirm each JN answers before the NN runs its cTime check
requireAllJnsReachable(sharedEditsUris);

static boolean isCTimeTimeout(IOException ioe) {
  return ioe.getCause() instanceof TimeoutException
      && ioe.getMessage().contains("getJournalCTime");
}

try {
  long ct = qjm.getJournalCTime();
} catch (IOException ioe) {
  if (isCTimeTimeout(ioe)) {
    retryAfterJnHealthCheck();
  } else {
    throw ioe;
  }
}

When it happens

Trigger: The upgrade-startup cTime check runs while a JournalNode is down, unreachable, or slow; any JN that fails to answer within dfs.qjm.operations.timeout triggers the timeout.

Common situations: A JN host is down when the NN starts with '-upgrade'; a JN's disk is slow to read its VERSION/segment metadata; network problems between NN and one JN; default 60s too tight for slow storage.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for doPreUpgrade() response
• Timed out waiting for doUpgrade() response
• Timed out waiting for doFinalize() response
• Timed out waiting for lockSharedStorage() response
• Timed out waiting for discardSegments() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20a8b55008fc40c1. Report an issue: GitHub.