apache/hadoop · error · FailoverFailedException

Got an IO exception

Error message

Got an IO exception

What it means

FailoverController.failover() wraps an IOException in FailoverFailedException('Got an IO exception') when HAServiceProtocolHelper.monitorHealth() — the pre-failover health check against the target (to-be-active) service — fails at the RPC layer. Failover is aborted before any state change because the target could not be reached or did not answer.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java:153

    if (!toSvcStatus.isReadyToBecomeActive()) {
      String notReadyReason = toSvcStatus.getNotReadyReason();
      if (!forceActive) {
        throw new FailoverFailedException(
            target + " is not ready to become active: " +
            notReadyReason);
      } else {
        LOG.warn("Service is not ready to become active, but forcing: {}",
            notReadyReason);
      }
    }

    try {
      HAServiceProtocolHelper.monitorHealth(toSvc, createReqInfo());
    } catch (HealthCheckFailedException hce) {
      throw new FailoverFailedException(
          "Can't failover to an unhealthy service", hce);
    } catch (IOException e) {
      throw new FailoverFailedException(
          "Got an IO exception", e);
    }
  }
  
  private StateChangeRequestInfo createReqInfo() {
    return new StateChangeRequestInfo(requestSource);
  }

  /**
   * Try to get the HA state of the node at the given address. This
   * function is guaranteed to be "quick" -- ie it has a short timeout
   * and no retries. Its only purpose is to avoid fencing a node that
   * has already restarted.
   */
  boolean tryGracefulFence(HAServiceTarget svc) {
    HAServiceProtocol proxy = null;
    try {
      proxy = svc.getProxy(gracefulFenceConf, gracefulFenceTimeout);

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the target service is up and reachable: 'hdfs haadmin -getServiceState <nsId><nnId>' or a direct RPC probe against its configured address.
2. Check DNS/firewall/connectivity from the node running the failover to the target's RPC address and port.
3. Confirm the client's Kerberos credentials (kinit) and hadoop.security.* settings match the cluster.
4. Retry the failover after connectivity is restored; failover to a down target cannot proceed.

Example fix

// before: failover attempted blindly
new FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);

// after: health-check the target's RPC first
HAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout
proxy.monitorHealth(); // throws IOException if unreachable -> fix connectivity first
new FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);

// Verify the target is reachable and healthy before failing over
HAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout, no retries
proxy.monitorHealth(); // throws IOException (unreachable) or HealthCheckFailedException (unhealthy)

try {
  new FailoverController(conf, RequestSource.REQUEST_BY_USER)
      .failover(fromSvc, toSvc, forceActive, forceFence);
} catch (FailoverFailedException ffe) {
  if (ffe.getCause() instanceof IOException) {
    // transport problem to the target: check address/firewall, then retry
  } else if (ffe.getCause() instanceof HealthCheckFailedException) {
    // target answered but is unhealthy: do not retry blindly
  }
}

When it happens

Trigger: Calling new FailoverController(conf, requestSource).failover(fromSvc, toSvc, forceActive, forceFence) or 'hdfs haadmin -failover' when the RPC to the target service's HAServiceProtocol address fails: target daemon down, wrong rpc-address/port, network partition, SASL/Kerberos handshake failure, or RPC timeout. Note this is the IOException branch — a health check that answered 'unhealthy' takes the HealthCheckFailedException branch instead.

Common situations: Target NameNode process stopped or still starting up; dfs.namenode.rpc-address typo between nodes; firewall blocking the RPC port; expired Kerberos tickets on the node running failover; long GC pause on the target making the health RPC time out.

Related errors

• Unable to failover to {}
• Unexpected ZooKeeper issue fetching active node info
• Cannot find any valid remote NN to service request!
• Transition from state {} to {} is not allowed.
• Retry " + retry + " times to read still exception: " + error

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f5504f8dde54462b. Report an issue: GitHub.