{"record":{"id":"112cdb9e867b8805","repo":"apache/hadoop","slug":"datanode-unreachable","errorCode":null,"errorMessage":"Datanode unreachable. {}","messagePattern":"Datanode unreachable\\. (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java","lineNumber":1148,"sourceCode":"\n    return exitCode;\n  }\n\n  /**\n   * Command to get balancer bandwidth for the given datanode. Usage: hdfs\n   * dfsadmin -getBalancerBandwidth {@literal <datanode_host:ipc_port>}\n   * @param argv List of of command line parameters.\n   * @param idx The index of the command that is being processed.\n   * @exception IOException\n   */\n  public int getBalancerBandwidth(String[] argv, int idx) throws IOException {\n    ClientDatanodeProtocol dnProxy = getDataNodeProxy(argv[idx]);\n    try {\n      long bandwidth = dnProxy.getBalancerBandwidth();\n      System.out.println(\"Balancer bandwidth is \" + bandwidth\n          + \" bytes per second.\");\n    } catch (IOException ioe) {\n      throw new IOException(\"Datanode unreachable. \" + ioe, ioe);\n    }\n    return 0;\n  }\n\n  /**\n   * Download the most recent fsimage from the name node, and save it to a local\n   * file in the given directory.\n   * \n   * @param argv\n   *          List of of command line parameters.\n   * @param idx\n   *          The index of the command that is being processed.\n   * @return an exit code indicating success or failure.\n   * @throws IOException\n   */\n  public int fetchImage(final String[] argv, final int idx) throws IOException {\n    Configuration conf = getConf();\n    final URL infoServer = DFSUtil.getInfoServer(","sourceCodeStart":1130,"sourceCodeEnd":1166,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java#L1130-L1166","documentation":"Implements `hdfs dfsadmin -getBalancerBandwidth <host:ipcPort>`. getDataNodeProxy builds a ClientDatanodeProtocol proxy to the datanode IPC address (dfs.datanode.ipc.address, default port 9867 in current Hadoop; 50020 in pre-3.x), and any IOException from the RPC — connect refused, timeout, SASL/Kerberos handshake failure — is wrapped as IOException('Datanode unreachable. <cause>') with the original as the cause.","triggerScenarios":"Passing the wrong port: the data-transfer port (9866/50010) or HTTP UI port (9864/50075) instead of the IPC port; datanode process down; host unreachable or firewalled; Kerberos principal mismatch between client config and the DN.","commonSituations":"Ops scripts that parse host from dfsadmin -report but hardcode a stale port; mixed-version clusters during rolling upgrade (port 50020 vs 9867); containers without the IPC port exposed; security enabled and dfs.datanode.kerberos.principal mis-set on the client side.","solutions":["Confirm the datanode and its IPC port: `hdfs dfsadmin -report` lists each DN's host:port; use that exact value (default 9867, older clusters 50020)","Probe reachability first: nc -vz <dn-host> 9867","Check the datanode process/log on that host if the port is closed","Unwrap the cause: the text after 'Datanode unreachable. ' tells whether it is a connection, timeout, or auth problem"],"exampleFix":"# before\n$ hdfs dfsadmin -getBalancerBandwidth dn1:9866\n# IOException: Datanode unreachable. ... (9866 is the data port)\n\n# after\n$ hdfs dfsadmin -getBalancerBandwidth dn1:9867   # dfs.datanode.ipc.address port\nBalancer bandwidth is 104857600 bytes per second.","handlingStrategy":"retry","validationCode":"// probe the datanode IPC port before building the admin proxy\ntry (Socket s = new Socket()) {\n  s.connect(new InetSocketAddress(dnHost, ipcPort), 2000); // dfs.datanode.ipc.address, default 9867\n} catch (IOException probe) {\n  return Result.datanodeDown(dnHost, probe); // skip the doomed RPC\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) { // 'Datanode unreachable. <cause>'\n  Throwable cause = e.getCause();\n  if (cause instanceof ConnectException || cause instanceof SocketTimeoutException) {\n    retryWithBackoff(); // transient: DN restarting or network blip\n  } else throw e;      // auth/version failure — fix config, do not retry\n}","preventionTips":["Always take host:port from hdfs dfsadmin -report instead of hardcoding","Remember the port map: 9867 IPC, 9866 data, 9864 HTTP (50020/50010/50075 on pre-3.x)","Probe the IPC port before looping over many datanodes in scripts"],"tags":["hdfs","dfsadmin","datanode","rpc","connection","balancer-bandwidth"],"backgroundTag":"connection-refused","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}