{"record":{"id":"6ad425fb40416e5c","repo":"apache/hadoop","slug":"timed-out-waiting-for-dofinalize-response","errorCode":null,"errorMessage":"Timed out waiting for doFinalize() response","messagePattern":"Timed out waiting for doFinalize\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":715,"sourceCode":"    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doUpgrade() response\");\n    }\n  }\n  \n  @Override\n  public void doFinalize() throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doFinalize();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doFinalize\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not finalize one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doFinalize() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doFinalize() response\");\n    }\n  }\n  \n  @Override\n  public boolean canRollBack(StorageInfo storage, StorageInfo prevStorage,\n      int targetLayoutVersion) throws IOException {\n    QuorumCall<AsyncLogger, Boolean> call = loggers.canRollBack(storage,\n        prevStorage, targetLayoutVersion);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"lockSharedStorage\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not check if roll back possible for\"\n            + \" one or more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can","sourceCodeStart":697,"sourceCodeEnd":733,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L697-L733","documentation":"QuorumJournalManager.doFinalize() fans out a doFinalize RPC to every JournalNode listed in dfs.namenode.shared.edits.dir and blocks until ALL of them answer (waitFor is called with min=max=loggers.size()). If the full set has not responded within timeoutMs (dfs.qjm.operations.timeout, default 60000ms), waitFor throws TimeoutException, which is wrapped in this IOException. It is thrown during NameNode upgrade finalization that touches the shared edits (QJM) storage.","triggerScenarios":"Invoking the QJM doFinalize() path (JournalManager upgrade-finalize, e.g. 'hdfs namenode -finalize'-class operations / upgrade finalization with shared edits) while at least one JournalNode is down, unreachable, or slower than dfs.qjm.operations.timeout; because the call waits for every logger, a single unresponsive JN triggers the timeout.","commonSituations":"One of the three JournalNodes is stopped or its host is down during finalize; a JN disk is failing or full so the local finalize RPC stalls; long GC pause on a JN; network partition between NN and one JN; retrying a finalize after a previous partial run; test/CI environments with a tight dfs.qjm.operations.timeout.","solutions":["Verify every JournalNode in dfs.namenode.shared.edits.dir is running and its RPC port is reachable from the NameNode host, then retry the finalize operation.","Inspect the slow/unreachable JN's logs for disk errors (full disk, bad disk, slow fsync) or GC pauses and fix the underlying slowness.","Raise dfs.qjm.operations.timeout above the slowest observed JN finalize time and retry (it defaults to 60000 ms).","If a JN is permanently lost, restore its journal directory from an identical copy of a healthy JN's directory (or fix the host) before retrying — doFinalize requires answers from all JNs, not a quorum."],"exampleFix":"// before\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>20000</value>\n</property>\n\n<!-- after: give slow JournalNodes time to answer doFinalize -->\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>120000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Preflight before doFinalize: every JN HTTP endpoint must answer\nfor (URI jn : sharedEditsUris) {\n  URL jstatus = new URL(\"http\", jn.getHost(), 8480, \"/jstatus\");\n  try (InputStream in = jstatus.openStream()) {\n    // JournalNode is alive and serving\n  } catch (IOException e) {\n    throw new IllegalStateException(\"JN not reachable before finalize: \" + jn, e);\n  }\n}","typeGuard":"static boolean isFinalizeTimeout(IOException ioe) {\n  return ioe.getCause() instanceof TimeoutException\n      && ioe.getMessage().contains(\"doFinalize\");\n}","tryCatchPattern":"try {\n  qjm.doFinalize();\n} catch (IOException ioe) {\n  if (ioe.getCause() instanceof TimeoutException) {\n    // transient: retry after confirming JN health, with backoff\n  } else {\n    throw ioe; // real per-JN failure, not a timeout\n  }\n}","preventionTips":["Monitor JournalNode availability (HTTP /jstatus) and disk health on every JN host before running upgrade finalize operations.","Set dfs.qjm.operations.timeout comfortably above the slowest JN's observed response time.","Keep all JournalNodes running during upgrade/rollback procedures — these code paths need every JN, not a quorum."],"tags":["qjm","journalnode","timeout","upgrade-finalize","rpc"],"backgroundTag":"rpc-quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}