apache/hadoop · error · FailoverFailedException

Got an IO exception

Error message

Got an IO exception

What it means

FailoverController.failover() wraps an IOException in FailoverFailedException('Got an IO exception') when HAServiceProtocolHelper.monitorHealth() — the pre-failover health check against the target (to-be-active) service — fails at the RPC layer. Failover is aborted before any state change because the target could not be reached or did not answer.

Solutions

1. Verify the target service is up and reachable: 'hdfs haadmin -getServiceState <nsId><nnId>' or a direct RPC probe against its configured address.
2. Check DNS/firewall/connectivity from the node running the failover to the target's RPC address and port.
3. Confirm the client's Kerberos credentials (kinit) and hadoop.security.* settings match the cluster.
4. Retry the failover after connectivity is restored; failover to a down target cannot proceed.

Example fix

// before: failover attempted blindly
new FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);

// after: health-check the target's RPC first
HAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout
proxy.monitorHealth(); // throws IOException if unreachable -> fix connectivity first
new FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);

// Verify the target is reachable and healthy before failing over
HAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout, no retries
proxy.monitorHealth(); // throws IOException (unreachable) or HealthCheckFailedException (unhealthy)

try {
  new FailoverController(conf, RequestSource.REQUEST_BY_USER)
      .failover(fromSvc, toSvc, forceActive, forceFence);
} catch (FailoverFailedException ffe) {
  if (ffe.getCause() instanceof IOException) {
    // transport problem to the target: check address/firewall, then retry
  } else if (ffe.getCause() instanceof HealthCheckFailedException) {
    // target answered but is unhealthy: do not retry blindly
  }
}

When it happens

Trigger: Calling new FailoverController(conf, requestSource).failover(fromSvc, toSvc, forceActive, forceFence) or 'hdfs haadmin -failover' when the RPC to the target service's HAServiceProtocol address fails: target daemon down, wrong rpc-address/port, network partition, SASL/Kerberos handshake failure, or RPC timeout. Note this is the IOException branch — a health check that answered 'unhealthy' takes the HealthCheckFailedException branch instead.

Common situations: Target NameNode process stopped or still starting up; dfs.namenode.rpc-address typo between nodes; firewall blocking the RPC port; expired Kerberos tickets on the node running failover; long GC pause on the target making the health RPC time out.

Related errors

• Cannot find any valid remote NN to service request!
• Transition from state
• Unable to failover to
• Unexpected ZooKeeper issue fetching active node info
• Bad argument

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f5504f8dde54462b. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java:153

    if (!toSvcStatus.isReadyToBecomeActive()) {
      String notReadyReason = toSvcStatus.getNotReadyReason();
      if (!forceActive) {
        throw new FailoverFailedException(
            target + " is not ready to become active: " +
            notReadyReason);
      } else {
        LOG.warn("Service is not ready to become active, but forcing: {}",
            notReadyReason);
      }
    }

    try {
      HAServiceProtocolHelper.monitorHealth(toSvc, createReqInfo());
    } catch (HealthCheckFailedException hce) {
      throw new FailoverFailedException(
          "Can't failover to an unhealthy service", hce);
    } catch (IOException e) {
      throw new FailoverFailedException(
          "Got an IO exception", e);
    }
  }
  
  private StateChangeRequestInfo createReqInfo() {
    return new StateChangeRequestInfo(requestSource);
  }

  /**
   * Try to get the HA state of the node at the given address. This
   * function is guaranteed to be "quick" -- ie it has a short timeout
   * and no retries. Its only purpose is to avoid fencing a node that
   * has already restarted.
   */
  boolean tryGracefulFence(HAServiceTarget svc) {
    HAServiceProtocol proxy = null;
    try {
      proxy = svc.getProxy(gracefulFenceConf, gracefulFenceTimeout);

View on GitHub (pinned to 2add963021)