{"record":{"id":"c244f8215be2483e","repo":"apache/hadoop","slug":"interrupted-waiting-for-format-response","errorCode":null,"errorMessage":"Interrupted waiting for format() response","messagePattern":"Interrupted waiting for format\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":258,"sourceCode":"    assert maxPromised >= 0;\n    \n    long myEpoch = maxPromised + 1;\n    Map<AsyncLogger, NewEpochResponseProto> resps =\n        loggers.waitForWriteQuorum(loggers.newEpoch(nsInfo, myEpoch),\n            newEpochTimeoutMs, \"newEpoch(\" + myEpoch + \")\");\n        \n    loggers.setEpoch(myEpoch);\n    return resps;\n  }\n  \n  @Override\n  public void format(NamespaceInfo nsInfo, boolean force) throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.format(nsInfo, force);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"format\");\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for format() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for format() response\");\n    }\n    \n    if (call.countExceptions() > 0) {\n      call.rethrowException(\"Could not format one or more JournalNodes\");\n    }\n  }\n\n  @Override\n  public boolean hasSomeData() throws IOException {\n    QuorumCall<AsyncLogger, Boolean> call =\n        loggers.isFormatted();\n\n    try {\n      call.waitFor(loggers.size(), 0, 0, timeoutMs, \"hasSomeData\");\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted while determining if JNs have data\");","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L240-L276","documentation":"QuorumJournalManager.format() sends format to every JournalNode and waits for all of them within dfs.qjm.operations.timeout. InterruptedException here means the formatting thread itself was cancelled (NN/tool shutdown), not that a JournalNode failed.","triggerScenarios":"hdfs namenode -format (or a QJM format API call) cancelled midway: JVM shutdown, Ctrl-C, or a caller-thread interrupt while waiting for JournalNode responses.","commonSituations":"Cluster bootstrap aborted by automation scripts; format step killed by a harness timeout; operators interrupting an HA format because one JN looked stuck.","solutions":["Identify what interrupted the formatter (shutdown hook, script kill) and rerun the format when it can run to completion.","Confirm all JournalNodes are up first so the rerun finishes within dfs.qjm.operations.timeout.","After an interrupted format, check each JN's formatted state before reusing the namespace, then re-run format -force if needed."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before formatting: confirm every JournalNode RPC port (default 8485) answers\nstatic boolean allJournalNodesReachable(List<InetSocketAddress> jns) {\n  for (InetSocketAddress a : jns) {\n    try (Socket s = new Socket()) {\n      s.connect(a, 2000);\n    } catch (IOException e) {\n      return false;\n    }\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  qjm.format(nsInfo, force);\n} catch (IOException e) {\n  if (Thread.currentThread().isInterrupted()\n      || (e.getMessage() != null && e.getMessage().contains(\"Interrupted waiting for format()\"))) {\n    Thread.currentThread().interrupt();\n    // cancelled mid-format: verify JN formatted state, then rerun format -force\n    verifyJournalNodeFormatState();\n    throw new IllegalStateException(\"format cancelled; rerun after cleanup\", e);\n  }\n  throw e;\n}","preventionTips":["Do not kill namenode -format midway; let bootstrap scripts run to completion with generous timeouts.","Verify all JournalNodes are reachable before starting format.","After any interrupted format, check each JN's on-disk state before reusing the namespace."],"tags":["hdfs","qjm","journal-node","format","interrupted","cluster-bootstrap"],"backgroundTag":"operation-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}