apache/hadoop · error · ServiceFailedException

Unable to become active. Local node did not get an opportuni

Error message

Unable to become active. Local node did not get an opportunity to do so from ZooKeeper, or the local node took too long to transition to active.

What it means

gracefulFailover() Phase 4: waitForActiveAttempt(timeout+60000) returned null while health stayed SERVICE_HEALTHY — the local ZKFC never got the opportunity to become active from ZooKeeper, or the transition took longer than the window; thrown as ServiceFailedException('Unable to become active. Local node did not get an opportunity to do so from ZooKeeper, or the local node took too long to transition to active.').

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:723

      activeNode != null : "Active node does not match any known remote node";

    // Phase 3b: ask the old active to yield
    otherZkfcs.add(cedeRemoteActive(activeNode, timeout));

    // Phase 4: wait for the normal election to make the local node
    // active.
    ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000, st);
    
    if (attempt == null) {
      // We didn't even make an attempt to become active.
      synchronized(this) {
        if (lastHealthState != State.SERVICE_HEALTHY) {
          throw new ServiceFailedException("Unable to become active. " +
            "Service became unhealthy while trying to failover.");          
        }
      }
      
      throw new ServiceFailedException("Unable to become active. " +
          "Local node did not get an opportunity to do so from ZooKeeper, " +
          "or the local node took too long to transition to active.");
    }

    // Phase 5. At this point, we made some attempt to become active. So we
    // can tell the old active to rejoin if it wants. This allows a quick
    // fail-back if we immediately crash.
    for (ZKFCProtocol zkfc : otherZkfcs) {
      zkfc.cedeActive(-1);
    }

    if (attempt.succeeded) {
      LOG.info("Successfully became active. " + attempt.status);
    } else {
      // Propagate failure
      String msg = "Failed to become active. " + attempt.status;
      throw new ServiceFailedException(msg);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check ZooKeeper health/latency and ZKFC logs on both nodes — did the local ZKFC rejoin the election after the cede?
2. Confirm the old active actually transitioned to standby (its logs show cedeActive handling); if it never ceded, fix its ZKFC.
3. Address local transition slowness: keep the standby's checkpoint state current, check JournalNode performance, look for a long tail of edits to replay.
4. Retry the failover with a larger timeout.

// Before failover: verify ZK latency is sane and old active ZKFC is responsive
// zkCli.sh -server zk:2181 create /probe x && delete /probe  (time it)
// hdfs haadmin -getServiceState on the old active must answer quickly

try {
  zkfc.gracefulFailover();
} catch (ServiceFailedException e) {
  if (e.getMessage().contains("did not get an opportunity")) {
    // election never reached local node: check ZK latency + old ZKFC cede,
    // then retry with a larger timeout after fixing the slow component
  }
}

When it happens

Trigger: The old active was asked to cede, but the local ZKFC never entered/won the election and never called becomeActive within timeout+60s: ZooKeeper latency or session issues, old active's ZKFC not actually ceding, or the local NameNode's transitionToActive exceeding the window (e.g. long checkpoint on transition).

Common situations: Overloaded/high-latency ZooKeeper ensemble; ha.zookeeper.session-timeout-ms too large so re-election is slow; old active ZKFC stuck; local NN slow to become active because it must checkpoint a large edit tail; timeout passed to gracefulFailover too small.

Related errors

• No other node is currently active.
• ZK Failover Controller failed: {}
• Unable to fence {}
• Unable to become active. Service became unhealthy while tryi
• Failed to become active. {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3223cd6e77fb8913. Report an issue: GitHub.