apache/hadoop · error · IOException

Failed to find datanode, suggest to check cluster health. ex

Error message

Failed to find datanode, suggest to check cluster health. excludeDatanodes={}

What it means

RouterWebHdfsMethods.redirectURI calls chooseDatanode(...) to pick a DataNode for the WebHDFS data-node redirect stage; when it returns null the Router throws this IOException. A null result means no usable DataNode was found: for OPEN/APPEND/GETFILECHECKSUM no replica host survived the excludes list (dead/decommissioned or excluded nodes), and for CREATE/other ops the Router's federation DataNode registry had no live DataNode to write to. The message embeds the client-supplied excludeDatanodes parameter for diagnosis.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java:435

   * @param excludeDatanodes Blocks to exclude.
   * @param parameters Other parameters.
   * @return Redirection URI.
   * @throws URISyntaxException If it cannot parse the URI.
   * @throws IOException If it cannot create the URI.
   */
  private URI redirectURI(final Router router, final UserGroupInformation ugi,
      final DelegationParam delegation, final UserParam username,
      final DoAsParam doAsUser, final String path, final HttpOpParam.Op op,
      final long openOffset, final String excludeDatanodes,
      final Param<?, ?>... parameters) throws URISyntaxException, IOException {
    if (!DFSUtil.isValidName(path)) {
      throw new InvalidPathException(path);
    }
    final DatanodeInfo dn =
        chooseDatanode(router, path, op, openOffset, excludeDatanodes);

    if (dn == null) {
      throw new IOException("Failed to find datanode, suggest to check cluster"
          + " health. excludeDatanodes=" + excludeDatanodes);
    }

    final String delegationQuery;
    if (!UserGroupInformation.isSecurityEnabled()) {
      // security disabled
      delegationQuery = Param.toSortedString("&", doAsUser, username);
    } else if (delegation.getValue() != null) {
      // client has provided a token
      delegationQuery = "&" + delegation;
    } else {
      // generate a token
      final Token<? extends TokenIdentifier> t = generateDelegationToken(
          ugi, ugi.getUserName());
      delegationQuery = "&delegation=" + t.encodeToUrlString();
    }

    final String redirectQuery = op.toQueryString() + delegationQuery

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the request with an empty/smaller excludeDatanodes so surviving replicas become eligible
2. Verify DataNode liveness and that DN reports reach the Router (Router admin/JMX DN membership, DataNode federation heartbeat config); fix the DN report pipeline if the list is empty
3. Check cluster health: dead/decommissioned DataNodes, under-replicated blocks, and NameNode block maps on the owning subcluster

catch (IOException e) {
  if (e.getMessage().startsWith("Failed to find datanode")) {
    // clear/shrink excludeDatanodes and retry; if persistent,
    // check DN liveness and Router DN membership reports
  }
}

When it happens

Trigger: WebHDFS OPEN/APPEND/GETFILECHECKSUM where all replica locations are in excludeDatanodes or reported dead; CREATE through Router WebHDFS when the Router has zero live DataNodes registered (DN reports not reaching the State Store); all blocks' DNs down or decommissioned.

Common situations: Clients retrying a read with the same excludeDatanodes list until every replica is excluded; DataNodes not sending federation reports (dfs.federation.router.datanode.* heartbeat not configured), so the Router never learns DN membership; small clusters where one dead DN removes the last replica.

Related errors

• Router is in startup mode
• {} is not supported
• File {} not found.
• Offset={} out of the range [0, {}); {}, path={}
• Usernames not matched: expecting null but name={name}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a0bb4347e22ae1b2. Report an issue: GitHub.