apache/hadoop · error · IOException

Failed to replace a bad datanode on the existing pipeline du

Error message

Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current={}, original={}). The current failed datanode replacement policy is {}, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.

What it means

Error "Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current={}, original={}). The current failed datanode replacement policy is {}, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration." thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:1452

          endOfBlockPacket.releaseBuffer(byteArrayManager);
        }
        endBlock();
      } else {
        initDataStreaming();
      }
    }

    return false;
  }

  void setHflush() {
    isHflushed = true;
  }

  private int findNewDatanode(final DatanodeInfo[] original
  ) throws IOException {
    if (nodes.length != original.length + 1) {
      throw new IOException(
          "Failed to replace a bad datanode on the existing pipeline "
              + "due to no more good datanodes being available to try. "
              + "(Nodes: current=" + Arrays.asList(nodes)
              + ", original=" + Arrays.asList(original) + "). "
              + "The current failed datanode replacement policy is "
              + dfsClient.dtpReplaceDatanodeOnFailure
              + ", and a client may configure this via '"
              + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
              + "' in its configuration.");
    }
    for(int i = 0; i < nodes.length; i++) {
      int j = 0;
      for(; j < original.length && !nodes[i].equals(original[j]); j++);
      if (j == original.length) {
        return i;
      }
    }
    throw new IOException("Failed: new datanode not found: nodes="

View on GitHub (pinned to 2add963021)

Solutions

1. Add more healthy DataNodes to the cluster so a replacement node is available for the pipeline.
2. Change dfs.client.block.write.replace-datanode-on-failure.policy (NEVER/DEFAULT/ALWAYS) to match cluster size; small clusters should use NEVER or DEFAULT.
3. Ensure failed DataNodes are decommissioned or recovered so enough nodes satisfy the replication factor.

When it happens

Trigger: A DataNode in the write pipeline fails and the client tries to replace it, but no additional healthy DataNodes are available. Whether the write fails depends on dfs.client.block.write.replace-datanode-on-failure.policy.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ba5120b2659c855f. Report an issue: GitHub.