{"record":{"id":"0724517581e0d363","repo":"apache/hadoop","slug":"timed-out-waiting-for-response-from-loggers","errorCode":null,"errorMessage":"Timed out waiting for response from loggers","messagePattern":"Timed out waiting for response from loggers","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":278,"sourceCode":"      throw new IOException(\"Timed out waiting for format() response\");\n    }\n    \n    if (call.countExceptions() > 0) {\n      call.rethrowException(\"Could not format one or more JournalNodes\");\n    }\n  }\n\n  @Override\n  public boolean hasSomeData() throws IOException {\n    QuorumCall<AsyncLogger, Boolean> call =\n        loggers.isFormatted();\n\n    try {\n      call.waitFor(loggers.size(), 0, 0, timeoutMs, \"hasSomeData\");\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted while determining if JNs have data\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for response from loggers\");\n    }\n    \n    if (call.countExceptions() > 0) {\n      call.rethrowException(\n          \"Unable to check if JNs are ready for formatting\");\n    }\n    \n    // If any of the loggers returned with a non-empty manifest, then\n    // we should prompt for format.\n    for (Boolean hasData : call.getResults().values()) {\n      if (hasData) {\n        return true;\n      }\n    }\n\n    // Otherwise, none were formatted, we can safely format.\n    return false;\n  }","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L260-L296","documentation":"hasSomeData() waits for all JournalNodes to report whether they hold edits, within dfs.qjm.operations.timeout. Any JournalNode that does not answer in time fails the check with this timeout, blocking the format-prompt path.","triggerScenarios":"One or more JournalNodes down, unreachable, or still starting when namenode -format checks for existing data; JN RPC port blocked from the NameNode host; dfs.qjm.operations.timeout too small relative to JN startup.","commonSituations":"HA cluster bootstrap with a JournalNode not yet started; firewall blocking port 8485; JN doing a long disk scan at startup while format runs.","solutions":["Ensure every JournalNode is running and reachable before formatting.","Raise dfs.qjm.operations.timeout and rerun if JournalNodes are merely slow to start.","Check JournalNode logs for errors answering the isFormatted RPC."],"exampleFix":"<!-- raise the QJM operation timeout before formatting -->\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>120000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Preflight for 'hdfs namenode -format': every JN must answer isFormatted in time\nstatic boolean journalNodesReady(List<InetSocketAddress> jns) {\n  for (InetSocketAddress a : jns) {\n    try (Socket s = new Socket()) {\n      s.connect(a, 2000);\n    } catch (IOException e) {\n      return false; // this JN will time out hasSomeData()\n    }\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  boolean hasData = qjm.hasSomeData();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Timed out waiting for response from loggers\")) {\n    // find and fix the silent JournalNode, then rerun the format flow\n    reportUnreachableJournalNodes();\n    throw new RuntimeException(\"hasSomeData timed out; JournalNode set unhealthy\", e);\n  }\n  throw e;\n}","preventionTips":["Wait for all JournalNodes to fully start (port 8485 listening) before running format.","Raise dfs.qjm.operations.timeout when JournalNodes boot slowly.","Check JN logs for isFormatted RPC errors (locked dirs, permissions) after any timeout."],"tags":["hdfs","qjm","journal-node","timeout","format","cluster-bootstrap"],"backgroundTag":"quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}