apache/hadoop · error · IOException

Datanode unreachable. {}

Error message

Datanode unreachable. {}

What it means

Implements `hdfs dfsadmin -getBalancerBandwidth <host:ipcPort>`. getDataNodeProxy builds a ClientDatanodeProtocol proxy to the datanode IPC address (dfs.datanode.ipc.address, default port 9867 in current Hadoop; 50020 in pre-3.x), and any IOException from the RPC — connect refused, timeout, SASL/Kerberos handshake failure — is wrapped as IOException('Datanode unreachable. <cause>') with the original as the cause.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:1148

    return exitCode;
  }

  /**
   * Command to get balancer bandwidth for the given datanode. Usage: hdfs
   * dfsadmin -getBalancerBandwidth {@literal <datanode_host:ipc_port>}
   * @param argv List of of command line parameters.
   * @param idx The index of the command that is being processed.
   * @exception IOException
   */
  public int getBalancerBandwidth(String[] argv, int idx) throws IOException {
    ClientDatanodeProtocol dnProxy = getDataNodeProxy(argv[idx]);
    try {
      long bandwidth = dnProxy.getBalancerBandwidth();
      System.out.println("Balancer bandwidth is " + bandwidth
          + " bytes per second.");
    } catch (IOException ioe) {
      throw new IOException("Datanode unreachable. " + ioe, ioe);
    }
    return 0;
  }

  /**
   * Download the most recent fsimage from the name node, and save it to a local
   * file in the given directory.
   * 
   * @param argv
   *          List of of command line parameters.
   * @param idx
   *          The index of the command that is being processed.
   * @return an exit code indicating success or failure.
   * @throws IOException
   */
  public int fetchImage(final String[] argv, final int idx) throws IOException {
    Configuration conf = getConf();
    final URL infoServer = DFSUtil.getInfoServer(

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm the datanode and its IPC port: `hdfs dfsadmin -report` lists each DN's host:port; use that exact value (default 9867, older clusters 50020)
2. Probe reachability first: nc -vz <dn-host> 9867
3. Check the datanode process/log on that host if the port is closed
4. Unwrap the cause: the text after 'Datanode unreachable. ' tells whether it is a connection, timeout, or auth problem

Example fix

# before
$ hdfs dfsadmin -getBalancerBandwidth dn1:9866
# IOException: Datanode unreachable. ... (9866 is the data port)

# after
$ hdfs dfsadmin -getBalancerBandwidth dn1:9867   # dfs.datanode.ipc.address port
Balancer bandwidth is 104857600 bytes per second.

// probe the datanode IPC port before building the admin proxy
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(dnHost, ipcPort), 2000); // dfs.datanode.ipc.address, default 9867
} catch (IOException probe) {
  return Result.datanodeDown(dnHost, probe); // skip the doomed RPC
}

catch (IOException e) { // 'Datanode unreachable. <cause>'
  Throwable cause = e.getCause();
  if (cause instanceof ConnectException || cause instanceof SocketTimeoutException) {
    retryWithBackoff(); // transient: DN restarting or network blip
  } else throw e;      // auth/version failure — fix config, do not retry
}

When it happens

Trigger: Passing the wrong port: the data-transfer port (9866/50010) or HTTP UI port (9864/50075) instead of the IPC port; datanode process down; host unreachable or firewalled; Kerberos principal mismatch between client config and the DN.

Common situations: Ops scripts that parse host from dfsadmin -report but hardcode a stale port; mixed-version clusters during rolling upgrade (port 50020 vs 9867); containers without the IPC port exposed; security enabled and dfs.datanode.kerberos.principal mis-set on the client side.

Related errors

• Replica was found but missing fields.
• Storage not yet initialized for {}
• Shutdown already in progress.
• UNKNOWN_KEY
• Failed to report bad block {} to namenode.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/112cdb9e867b8805. Report an issue: GitHub.