apache/hadoop · error · ServiceFailedException

Unable to become active. Service became unhealthy while tryi

Error message

Unable to become active. Service became unhealthy while trying to failover.

What it means

In gracefulFailover() Phase 4, if waitForActiveAttempt returns no ActiveAttemptRecord within timeout+60000 ms and lastHealthState != SERVICE_HEALTHY, the ZKFC throws ServiceFailedException('Unable to become active. Service became unhealthy while trying to failover.') — the local service's health degraded during the takeover window.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:718

      }
      otherZkfcs.add(cedeRemoteActive(remote, timeout));
    }

    assert
      activeNode != null : "Active node does not match any known remote node";

    // Phase 3b: ask the old active to yield
    otherZkfcs.add(cedeRemoteActive(activeNode, timeout));

    // Phase 4: wait for the normal election to make the local node
    // active.
    ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000, st);
    
    if (attempt == null) {
      // We didn't even make an attempt to become active.
      synchronized(this) {
        if (lastHealthState != State.SERVICE_HEALTHY) {
          throw new ServiceFailedException("Unable to become active. " +
            "Service became unhealthy while trying to failover.");          
        }
      }
      
      throw new ServiceFailedException("Unable to become active. " +
          "Local node did not get an opportunity to do so from ZooKeeper, " +
          "or the local node took too long to transition to active.");
    }

    // Phase 5. At this point, we made some attempt to become active. So we
    // can tell the old active to rejoin if it wants. This allows a quick
    // fail-back if we immediately crash.
    for (ZKFCProtocol zkfc : otherZkfcs) {
      zkfc.cedeActive(-1);
    }

    if (attempt.succeeded) {
      LOG.info("Successfully became active. " + attempt.status);

View on GitHub (pinned to 2add963021)

Solutions

Check the local NameNode's health and logs for whatever made the health monitor leave SERVICE_HEALTHY (crash, OOM, disk, checkpoint failure).
Run 'hdfs haadmin -checkHealth <nnId>' once it responds.
Fix the health issue, then retry the graceful failover.
If the old active already ceded, the cluster has no active — verify with getAllServiceState and force/allow election.

Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight the target the same way the ZKFC does
zkfc.checkEligibleForFailover(); // throws ServiceFailedException if not healthy/observer
// or externally: hdfs haadmin -checkHealth <nnId>

Try / catch

try {
  zkfc.gracefulFailover();
} catch (ServiceFailedException e) {
  if (e.getMessage().contains("Service became unhealthy")) {
    // inspect target NN health/logs; do not retry until health is green;
    // also confirm whether the old active needs to rejoin (cedeActive(-1))
  }
}

Prevention

Run 'hdfs haadmin -checkHealth' on the failover target before every manual failover.
Alert on health-monitor state changes on all HA nodes so you never failover into an unhealthy node.
Keep NN memory/disks healthy; most mid-failover health losses are OOM or disk-full events.

When it happens

Trigger: The local node's health monitor reported a state other than SERVICE_HEALTHY (health check RPC failed, checkHealth threw, monitor shut down) between the cede request to the old active and the wait for the local transition attempt.

Common situations: NameNode crashed or OOM-killed mid-failover; transient health-check RPC failures to the local NN; disk full or edits-dir errors turning the NN unhealthy exactly during failover; NN restarted during the operation.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c17023b529345e80. Report an issue: GitHub.