apache/hadoop · error · IOException

Failed to add a datanode. Response status: {}

Error message

Failed to add a datanode. Response status: {}

What it means

During write-pipeline error recovery, DataStreamer can ask a datanode to TRANSFER an existing block to a newly added datanode (DatanodeProtocol TRANSFER_BLOCK, sent by TransferBuilder in DataStreamer.java:191). The client parses the BlockOpResponseProto and, unless the status is SUCCESS, throws IOException('Failed to add a datanode. Response status: ...') — the incremental repair of the pipeline was refused or failed on the datanode side.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:191

      out = new DataOutputStream(new BufferedOutputStream(unbufOut,
          DFSUtilClient.getSmallBufferSize(dfsClient.getConfiguration())));
      in = new DataInputStream(unbufIn);
    }

    void sendTransferBlock(final DatanodeInfo[] targets,
        final StorageType[] targetStorageTypes,
        final String[] targetStorageIDs,
        final Token<BlockTokenIdentifier> blockToken) throws IOException {
      //send the TRANSFER_BLOCK request
      new Sender(out).transferBlock(block.getCurrentBlock(), blockToken,
          dfsClient.clientName, targets, targetStorageTypes,
          targetStorageIDs);
      out.flush();
      //ack
      BlockOpResponseProto transferResponse = BlockOpResponseProto
          .parseFrom(PBHelperClient.vintPrefixed(in));
      if (SUCCESS != transferResponse.getStatus()) {
        throw new IOException("Failed to add a datanode. Response status: "
            + transferResponse.getStatus());
      }
    }

    @Override
    public void close() throws IOException {
      IOUtils.closeStream(in);
      IOUtils.closeStream(out);
      IOUtils.closeSocket(sock);
    }
  }

  static class BlockToWrite {
    private ExtendedBlock currentBlock;

    BlockToWrite(ExtendedBlock block) {
      setCurrentBlock(block);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the datanode logs for the nodes involved at the failure timestamp — the DN-side reason for the non-success status is logged there.
2. Verify target datanode health/capacity with hdfs dfsadmin -report (look for full or read-only volumes).
3. Retry the write: transient transfer failures usually clear on the next recovery attempt, which picks different nodes.
4. If replacement attempts themselves destabilize writes on a fragile cluster, review dfs.client.block.write.replace-datanode-on-failure.policy.

// this fires inside DataStreamer's internal recovery; at the app level catch
// IOException from write/flush and retry the whole write with backoff
try {
  out.write(chunk); out.hflush();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to add a datanode")) {
    // datanode-side transfer refused: usually transient; retry after backoff
    retryWriteWithBackoff(chunk);
  } else {
    throw e;
  }
}

When it happens

Trigger: Pipeline recovery tries to add a replacement datanode (dfs.client.block.write.replace-datanode-on-failure policy default) and the involved datanode returns a non-success transfer status: source replica unavailable, target disk full or storage-type mismatch, or transfer thread failure on the DN.

Common situations: Full disks on the replacement datanode; source replica missing/moved mid-transfer; storage-type constraints (e.g., ARCHIVE-only nodes rejecting a DISK replica); DN overload during recovery windows.

Related errors

• Data streamers failed while creating new block streams: {}.
• Unable to create new block.{}
• Version Mismatch (Expected: {}, Received: {} )
• Unknown op {} in data stream
• Cannot create a secured connection if DataNode listens on un

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb42f08cf01b4ce1. Report an issue: GitHub.