{"record":{"id":"3d7818191902cac4","repo":"apache/hadoop","slug":"failed-to-start-active-services","errorCode":null,"errorMessage":"Failed to start active services","messagePattern":"Failed to start active services","errorType":"exception","errorClass":"ServiceFailedException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ActiveState.java","lineNumber":63,"sourceCode":"  public boolean shouldPopulateReplQueues() {\n    return true;\n  }\n  \n  @Override\n  public void setState(HAContext context, HAState s) throws ServiceFailedException {\n    if (s == NameNode.STANDBY_STATE) {\n      setStateInternal(context, s);\n      return;\n    }\n    super.setState(context, s);\n  }\n\n  @Override\n  public void enterState(HAContext context) throws ServiceFailedException {\n    try {\n      context.startActiveServices();\n    } catch (IOException e) {\n      throw new ServiceFailedException(\"Failed to start active services\", e);\n    }\n  }\n\n  @Override\n  public void exitState(HAContext context) throws ServiceFailedException {\n    try {\n      context.stopActiveServices();\n    } catch (IOException e) {\n      throw new ServiceFailedException(\"Failed to stop active services\", e);\n    }\n  }\n\n}\n","sourceCodeStart":45,"sourceCodeEnd":77,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ActiveState.java#L45-L77","documentation":"In the HA state machine, ActiveState.enterState() delegates to HAContext.startActiveServices() and wraps any IOException as ServiceFailedException('Failed to start active services'). The real failure is always the cause chain: common services started there include edit-log/journal initialization, RPC and lease/monitor threads.","triggerScenarios":"'hdfs haadmin -transitionToActive', ZKFC-triggered failover, or requestStateTransition to ACTIVE when startActiveServices() throws — e.g., shared edits (QJM) unreachable or unformatted, journal manager init failure, port already in use, resource limit errors.","commonSituations":"Standby promoted before -bootstrapStandby so the shared-edits dir disagrees; QJM quorum down during failover; leftover process holding the RPC port; a half-configured HA pair after a config rollout.","solutions":["Inspect the ServiceFailedException cause chain in the NameNode log — the fix is always for the wrapped IOException, not this wrapper","If shared edits is the cause: run 'hdfs namenode -bootstrapStandby' with a healthy active, or format/repair QJM","Free conflicting resources (ports, locks) and retry the transition","As a last resort restart the NameNode and re-attempt transition to ACTIVE"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight the usual root causes before transitioning\nif (!HAUtil.usesSharedEditsDir(conf)) throw new IOException(\"no shared edits\");\n// verify QJM quorum reachable:\nfor (InetSocketAddress a : getJournalAddresses(conf)) {\n  try (Socket s = new Socket()) { s.connect(a, 3000); }\n  catch (IOException ioe) { throw new IOException(\"journal down: \" + a, ioe); }\n}","typeGuard":null,"tryCatchPattern":"try {\n  haAdmin.transitionToActive(nn);\n} catch (ServiceFailedException e) { // \"Failed to start active services\"\n  Throwable root = e.getCause();          // the real reason lives here\n  LOG.error(\"Active transition failed: {}\", root, root);\n  // leave/return the node to STANDBY; fix cause (shared edits, ports), then retry\n}","preventionTips":["Always run -bootstrapStandby on a new standby before first transition to ACTIVE","Monitor QJM/journal availability so failover never fires into a dead quorum","Rehearse manual failover in maintenance windows instead of first trying under incident pressure"],"tags":["hdfs","namenode","ha","state-transition","service-failed","failover"],"backgroundTag":"service-startup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}