apache/hadoop · critical · ServiceFailedException

Failed to start active services

Error message

Failed to start active services

What it means

In the HA state machine, ActiveState.enterState() delegates to HAContext.startActiveServices() and wraps any IOException as ServiceFailedException('Failed to start active services'). The real failure is always the cause chain: common services started there include edit-log/journal initialization, RPC and lease/monitor threads.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ActiveState.java:63

  public boolean shouldPopulateReplQueues() {
    return true;
  }
  
  @Override
  public void setState(HAContext context, HAState s) throws ServiceFailedException {
    if (s == NameNode.STANDBY_STATE) {
      setStateInternal(context, s);
      return;
    }
    super.setState(context, s);
  }

  @Override
  public void enterState(HAContext context) throws ServiceFailedException {
    try {
      context.startActiveServices();
    } catch (IOException e) {
      throw new ServiceFailedException("Failed to start active services", e);
    }
  }

  @Override
  public void exitState(HAContext context) throws ServiceFailedException {
    try {
      context.stopActiveServices();
    } catch (IOException e) {
      throw new ServiceFailedException("Failed to stop active services", e);
    }
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the ServiceFailedException cause chain in the NameNode log — the fix is always for the wrapped IOException, not this wrapper
2. If shared edits is the cause: run 'hdfs namenode -bootstrapStandby' with a healthy active, or format/repair QJM
3. Free conflicting resources (ports, locks) and retry the transition
4. As a last resort restart the NameNode and re-attempt transition to ACTIVE

// pre-flight the usual root causes before transitioning
if (!HAUtil.usesSharedEditsDir(conf)) throw new IOException("no shared edits");
// verify QJM quorum reachable:
for (InetSocketAddress a : getJournalAddresses(conf)) {
  try (Socket s = new Socket()) { s.connect(a, 3000); }
  catch (IOException ioe) { throw new IOException("journal down: " + a, ioe); }
}

try {
  haAdmin.transitionToActive(nn);
} catch (ServiceFailedException e) { // "Failed to start active services"
  Throwable root = e.getCause();          // the real reason lives here
  LOG.error("Active transition failed: {}", root, root);
  // leave/return the node to STANDBY; fix cause (shared edits, ports), then retry
}

When it happens

Trigger: 'hdfs haadmin -transitionToActive', ZKFC-triggered failover, or requestStateTransition to ACTIVE when startActiveServices() throws — e.g., shared edits (QJM) unreachable or unformatted, journal manager init failure, port already in use, resource limit errors.

Common situations: Standby promoted before -bootstrapStandby so the shared-edits dir disagrees; QJM quorum down during failover; leftover process holding the RPC port; a half-configured HA pair after a config rollout.

Related errors

• Cannot transition from 'observer' to 'active'
• {getRole()} still not leave safemode
• Cannot transition from 'active' to 'observer'
• Failed to stop active services
• Block pool {bpid} has not recognized an active NN

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d7818191902cac4. Report an issue: GitHub.