{"record":{"id":"9e9db62387f5656c","repo":"apache/hadoop","slug":"does-not-support-multiple-targets","errorCode":null,"errorMessage":"{} does not support multiple targets {}","messagePattern":"(.+?) does not support multiple targets (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java","lineNumber":737,"sourceCode":"    // To support older clients, we don't pass in empty storageIds\n    final int nsi = targetStorageIds.length;\n    final String[] storageIds;\n    if (nsi > 0) {\n      storageIds = new String[nsi + 1];\n      storageIds[0] = storageId;\n      if (targetStorageTypes.length > 0) {\n        System.arraycopy(targetStorageIds, 0, storageIds, 1, nsi);\n      }\n    } else {\n      storageIds = new String[0];\n    }\n    checkAccess(replyOut, isClient, block, blockToken, Op.WRITE_BLOCK,\n        BlockTokenIdentifier.AccessMode.WRITE,\n        storageTypes, storageIds);\n\n    // check single target for transfer-RBW/Finalized\n    if (isTransfer && targets.length > 0) {\n      throw new IOException(stage + \" does not support multiple targets \"\n          + Arrays.asList(targets));\n    }\n\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"opWriteBlock: stage={}, clientname={}\\n  \" +\n              \"block  ={}, newGs={}, bytesRcvd=[{}, {}]\\n  \" +\n              \"targets={}; pipelineSize={}, srcDataNode={}, pinning={}\",\n          stage, clientname, block, latestGenerationStamp, minBytesRcvd,\n          maxBytesRcvd, Arrays.asList(targets), pipelineSize, srcDataNode,\n          pinning);\n      LOG.debug(\"isDatanode={}, isClient={}, isTransfer={}\",\n          isDatanode, isClient, isTransfer);\n      LOG.debug(\"writeBlock receive buf size {} tcp no delay {}\",\n          peer.getReceiveBufferSize(), peer.getTcpNoDelay());\n    }\n\n    // We later mutate block's generation stamp and length, but we need to\n    // forward the original version of the block to downstream mirrors, so","sourceCodeStart":719,"sourceCodeEnd":755,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java#L719-L755","documentation":"IOException from DataXceiver.opWriteBlock when a block write arrives with a pipeline stage that is a datanode-to-datanode transfer (PIPELINE_SETUP_TRANSFER / TRANSFER_RBW / TRANSFER_FINALIZED, i.e. isTransfer == true) and the request carries more than zero downstream targets. Block transfers between DataNodes (used by rebalancing, stripe/mover operations, and replication recovery moves) move data to exactly one destination; a real client append/create pipeline is the only mode allowed to specify multiple targets.","triggerScenarios":"A WRITE_BLOCK request with stage == PIPELINE_SETUP_TRANSFER (or TRANSFER_RBW/TRANSFER_FINALIZED) whose targets array is non-empty. Produced by a misbehaving or version-mismatched upstream DataNode/client that populates pipeline targets on a transfer request, e.g.dfsclient replication code sending a chain of targets where the transfer protocol expects none.","commonSituations":"Interoperability bugs between DataNode versions during a rolling upgrade (older sender includes targets the newer receiver rejects); patched or third-party clients (Erasure coding prototypes, balancer forks) that reuse the client write path for transfers; extremely rare on stock matching versions.","solutions":["Check for version mismatch between the source DataNode initiating the transfer and the rejecting DataNode; complete the rolling upgrade so both run the same release","Capture the stage and target list from the DataNode debug log (the LOG.debug right after the check prints stage, targets, pipelineSize) and compare against the sender's code path","If a custom client/balancer is in play, fix it to send an empty targets array for TRANSFER_RBW/TRANSFER_FINALIZED/PIPELINE_SETUP_TRANSFER stages","Retry the operation (balancer/mover) once versions are aligned; single-target transfer is retried per-block automatically"],"exampleFix":"// before (custom client / DN sending a transfer): targets populated -> DN throws\n//   opWriteBlock(stage=TRANSFER_RBW, targets=[dn2, dn3])\nnew Sender(out).writeBlock(block, STORAGE_TYPE, STORAGE_ID, tokens,\n    \"source-datanode\", 0, BLOCK_SIZE, 0, 0,\n    new DatanodeInfo[]{dn2, dn3},      // WRONG on transfer\n    new StorageType[]{DISK, DISK}, null, Stage.TRANSFER_RBW, ...);\n\n// after: transfers carry no downstream targets\nnew Sender(out).writeBlock(block, STORAGE_TYPE, STORAGE_ID, tokens,\n    \"source-datanode\", 0, BLOCK_SIZE, 0, 0,\n    new DatanodeInfo[]{},               // transfer = zero targets\n    new StorageType[]{}, null, Stage.TRANSFER_RBW, ...);","handlingStrategy":"try-catch","validationCode":"// Sender-side guard: only client stages may carry pipeline targets\nboolean isTransfer = stage == BlockConstructionStage.TRANSFER_RBW\n    || stage == BlockConstructionStage.TRANSFER_FINALIZED\n    || stage == BlockConstructionStage.PIPELINE_SETUP_TRANSFER;\nDatanodeInfo[] targetsToSend = isTransfer ? new DatanodeInfo[0] : targets;\n// pass targetsToSend to Sender.writeBlock(...)","typeGuard":null,"tryCatchPattern":"try {\n  sendWriteBlock(stage, targets);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"does not support multiple targets\")) {\n    // protocol bug or version mismatch: retry the single-target transfer,\n    // and align sender/receiver Hadoop versions\n    sendWriteBlock(stage, new DatanodeInfo[0]);\n  } else { throw e; }\n}","preventionTips":["Keep DataNode versions uniform, especially during rolling upgrades, so transfer senders and receivers agree on target semantics","In custom DFS clients, assert targets.length == 0 for TRANSFER_* / PIPELINE_SETUP_TRANSFER stages before writing","Enable DataXceiver debug logging when developing against the data transfer protocol to see the exact stage/targets a peer rejects"],"tags":["hdfs","dataxceiver","write-block","block-transfer","protocol","pipeline"],"backgroundTag":"protocol-contract-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}