{"record":{"id":"fb42f08cf01b4ce1","repo":"apache/hadoop","slug":"failed-to-add-a-datanode-response-status","errorCode":null,"errorMessage":"Failed to add a datanode. Response status: {}","messagePattern":"Failed to add a datanode\\. Response status: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java","lineNumber":191,"sourceCode":"      out = new DataOutputStream(new BufferedOutputStream(unbufOut,\n          DFSUtilClient.getSmallBufferSize(dfsClient.getConfiguration())));\n      in = new DataInputStream(unbufIn);\n    }\n\n    void sendTransferBlock(final DatanodeInfo[] targets,\n        final StorageType[] targetStorageTypes,\n        final String[] targetStorageIDs,\n        final Token<BlockTokenIdentifier> blockToken) throws IOException {\n      //send the TRANSFER_BLOCK request\n      new Sender(out).transferBlock(block.getCurrentBlock(), blockToken,\n          dfsClient.clientName, targets, targetStorageTypes,\n          targetStorageIDs);\n      out.flush();\n      //ack\n      BlockOpResponseProto transferResponse = BlockOpResponseProto\n          .parseFrom(PBHelperClient.vintPrefixed(in));\n      if (SUCCESS != transferResponse.getStatus()) {\n        throw new IOException(\"Failed to add a datanode. Response status: \"\n            + transferResponse.getStatus());\n      }\n    }\n\n    @Override\n    public void close() throws IOException {\n      IOUtils.closeStream(in);\n      IOUtils.closeStream(out);\n      IOUtils.closeSocket(sock);\n    }\n  }\n\n  static class BlockToWrite {\n    private ExtendedBlock currentBlock;\n\n    BlockToWrite(ExtendedBlock block) {\n      setCurrentBlock(block);\n    }","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L173-L209","documentation":"During write-pipeline error recovery, DataStreamer can ask a datanode to TRANSFER an existing block to a newly added datanode (DatanodeProtocol TRANSFER_BLOCK, sent by TransferBuilder in DataStreamer.java:191). The client parses the BlockOpResponseProto and, unless the status is SUCCESS, throws IOException('Failed to add a datanode. Response status: ...') — the incremental repair of the pipeline was refused or failed on the datanode side.","triggerScenarios":"Pipeline recovery tries to add a replacement datanode (dfs.client.block.write.replace-datanode-on-failure policy default) and the involved datanode returns a non-success transfer status: source replica unavailable, target disk full or storage-type mismatch, or transfer thread failure on the DN.","commonSituations":"Full disks on the replacement datanode; source replica missing/moved mid-transfer; storage-type constraints (e.g., ARCHIVE-only nodes rejecting a DISK replica); DN overload during recovery windows.","solutions":["Check the datanode logs for the nodes involved at the failure timestamp — the DN-side reason for the non-success status is logged there.","Verify target datanode health/capacity with hdfs dfsadmin -report (look for full or read-only volumes).","Retry the write: transient transfer failures usually clear on the next recovery attempt, which picks different nodes.","If replacement attempts themselves destabilize writes on a fragile cluster, review dfs.client.block.write.replace-datanode-on-failure.policy."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// this fires inside DataStreamer's internal recovery; at the app level catch\n// IOException from write/flush and retry the whole write with backoff\ntry {\n  out.write(chunk); out.hflush();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Failed to add a datanode\")) {\n    // datanode-side transfer refused: usually transient; retry after backoff\n    retryWriteWithBackoff(chunk);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Keep datanode disks below capacity and monitor volumes going read-only — TRANSFER_BLOCK to a full DN fails.","Review dfs.client.block.write.replace-datanode-on-failure.* on clusters where replacement attempts repeatedly fail.","Correlate with datanode logs: the response status in the message maps to a DN-side reason (source missing, disk full, storage type)."],"tags":["hdfs","write","pipeline-recovery","datanode","block-transfer"],"backgroundTag":"datanode-block-transfer-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}