{"record":{"id":"c96ea9aa8b867a4d","repo":"apache/hadoop","slug":"timed-out-waiting-for-format-response","errorCode":null,"errorMessage":"Timed out waiting for format() response","messagePattern":"Timed out waiting for format\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":260,"sourceCode":"    long myEpoch = maxPromised + 1;\n    Map<AsyncLogger, NewEpochResponseProto> resps =\n        loggers.waitForWriteQuorum(loggers.newEpoch(nsInfo, myEpoch),\n            newEpochTimeoutMs, \"newEpoch(\" + myEpoch + \")\");\n        \n    loggers.setEpoch(myEpoch);\n    return resps;\n  }\n  \n  @Override\n  public void format(NamespaceInfo nsInfo, boolean force) throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.format(nsInfo, force);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"format\");\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for format() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for format() response\");\n    }\n    \n    if (call.countExceptions() > 0) {\n      call.rethrowException(\"Could not format one or more JournalNodes\");\n    }\n  }\n\n  @Override\n  public boolean hasSomeData() throws IOException {\n    QuorumCall<AsyncLogger, Boolean> call =\n        loggers.isFormatted();\n\n    try {\n      call.waitFor(loggers.size(), 0, 0, timeoutMs, \"hasSomeData\");\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted while determining if JNs have data\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for response from loggers\");","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L242-L278","documentation":"format() requires every JournalNode to acknowledge (QuorumCall.waitFor with loggers.size() successes) within dfs.qjm.operations.timeout. A single unreachable or slow JournalNode therefore fails the whole format with this timeout.","triggerScenarios":"Running hdfs namenode -format while any JournalNode in the quorum is down, firewalled on its RPC port (default 8485), or too slow to answer; dfs.qjm.operations.timeout set below JN response time.","commonSituations":"First bootstrap of an HA cluster before all JournalNodes are started; security groups blocking NN to JN traffic; JN disk or startup problems during initial formatting.","solutions":["Start every JournalNode and verify from the NameNode host that each JN RPC port answers.","Raise dfs.qjm.operations.timeout if JournalNodes are slow, then rerun the format.","Check JournalNode logs for errors handling the format RPC (locked edits directories, permissions).","Once the full JN set is healthy, rerun format (with -force) because a partially formatted quorum must be redone."],"exampleFix":"<!-- raise the QJM operation timeout before formatting -->\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>120000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Preflight: every JournalNode must answer before 'hdfs namenode -format'\n// (format requires ALL JNs, not just a majority)\nfor (String jn : journalNodeHosts) {\n  try (Socket s = new Socket()) {\n    s.connect(new InetSocketAddress(jn, 8485), 2000);\n  } catch (IOException e) {\n    throw new IllegalStateException(\"JournalNode not reachable: \" + jn, e);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  qjm.format(nsInfo, force);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Timed out waiting for format()\")) {\n    // identify the non-responding JN, fix/start it, then rerun format\n    reportUnreachableJournalNodes();\n    throw new RuntimeException(\"format failed: not all JournalNodes answered in time\", e);\n  }\n  throw e;\n}","preventionTips":["Start all JournalNodes before any NameNode format; format requires every JN.","Open JN RPC port 8485 in firewall rules between NN and JNs at bootstrap time.","Size dfs.qjm.operations.timeout to your slowest JN's startup and disk speed."],"tags":["hdfs","qjm","journal-node","format","timeout","cluster-bootstrap"],"backgroundTag":"quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}