{"record":{"id":"f5504f8dde54462b","repo":"apache/hadoop","slug":"got-an-io-exception","errorCode":null,"errorMessage":"Got an IO exception","messagePattern":"Got an IO exception","errorType":"exception","errorClass":"FailoverFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java","lineNumber":153,"sourceCode":"    if (!toSvcStatus.isReadyToBecomeActive()) {\n      String notReadyReason = toSvcStatus.getNotReadyReason();\n      if (!forceActive) {\n        throw new FailoverFailedException(\n            target + \" is not ready to become active: \" +\n            notReadyReason);\n      } else {\n        LOG.warn(\"Service is not ready to become active, but forcing: {}\",\n            notReadyReason);\n      }\n    }\n\n    try {\n      HAServiceProtocolHelper.monitorHealth(toSvc, createReqInfo());\n    } catch (HealthCheckFailedException hce) {\n      throw new FailoverFailedException(\n          \"Can't failover to an unhealthy service\", hce);\n    } catch (IOException e) {\n      throw new FailoverFailedException(\n          \"Got an IO exception\", e);\n    }\n  }\n  \n  private StateChangeRequestInfo createReqInfo() {\n    return new StateChangeRequestInfo(requestSource);\n  }\n\n  /**\n   * Try to get the HA state of the node at the given address. This\n   * function is guaranteed to be \"quick\" -- ie it has a short timeout\n   * and no retries. Its only purpose is to avoid fencing a node that\n   * has already restarted.\n   */\n  boolean tryGracefulFence(HAServiceTarget svc) {\n    HAServiceProtocol proxy = null;\n    try {\n      proxy = svc.getProxy(gracefulFenceConf, gracefulFenceTimeout);","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java#L135-L171","documentation":"FailoverController.failover() wraps an IOException in FailoverFailedException('Got an IO exception') when HAServiceProtocolHelper.monitorHealth() — the pre-failover health check against the target (to-be-active) service — fails at the RPC layer. Failover is aborted before any state change because the target could not be reached or did not answer.","triggerScenarios":"Calling new FailoverController(conf, requestSource).failover(fromSvc, toSvc, forceActive, forceFence) or 'hdfs haadmin -failover' when the RPC to the target service's HAServiceProtocol address fails: target daemon down, wrong rpc-address/port, network partition, SASL/Kerberos handshake failure, or RPC timeout. Note this is the IOException branch — a health check that answered 'unhealthy' takes the HealthCheckFailedException branch instead.","commonSituations":"Target NameNode process stopped or still starting up; dfs.namenode.rpc-address typo between nodes; firewall blocking the RPC port; expired Kerberos tickets on the node running failover; long GC pause on the target making the health RPC time out.","solutions":["Verify the target service is up and reachable: 'hdfs haadmin -getServiceState <nsId><nnId>' or a direct RPC probe against its configured address.","Check DNS/firewall/connectivity from the node running the failover to the target's RPC address and port.","Confirm the client's Kerberos credentials (kinit) and hadoop.security.* settings match the cluster.","Retry the failover after connectivity is restored; failover to a down target cannot proceed."],"exampleFix":"// before: failover attempted blindly\nnew FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);\n\n// after: health-check the target's RPC first\nHAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout\nproxy.monitorHealth(); // throws IOException if unreachable -> fix connectivity first\nnew FailoverController(conf, RequestSource.REQUEST_BY_USER).failover(fromSvc, toSvc, false, false);","handlingStrategy":"try-catch","validationCode":"// Verify the target is reachable and healthy before failing over\nHAServiceProtocol proxy = toSvc.getProxy(conf, 5000); // short timeout, no retries\nproxy.monitorHealth(); // throws IOException (unreachable) or HealthCheckFailedException (unhealthy)","typeGuard":null,"tryCatchPattern":"try {\n  new FailoverController(conf, RequestSource.REQUEST_BY_USER)\n      .failover(fromSvc, toSvc, forceActive, forceFence);\n} catch (FailoverFailedException ffe) {\n  if (ffe.getCause() instanceof IOException) {\n    // transport problem to the target: check address/firewall, then retry\n  } else if (ffe.getCause() instanceof HealthCheckFailedException) {\n    // target answered but is unhealthy: do not retry blindly\n  }\n}","preventionTips":["Monitor RPC reachability of every HA service address continuously so a dead target is known before failover is attempted.","Keep dfs.namenode.rpc-address identical in meaning across nodes' configs and validate configs with a config-diff tool.","Automate failover through the ZKFC rather than manual FailoverController calls; it re-checks health as part of election."],"tags":["hadoop","high-availability","failover","rpc","network"],"backgroundTag":"rpc-connection-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}