{"record":{"id":"baea183c7ded0068","repo":"apache/hadoop","slug":"unable-to-create-new-block-baea18","errorCode":null,"errorMessage":"Unable to create new block.{}","messagePattern":"Unable to create new block\\.(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java","lineNumber":113,"sourceCode":"    LocatedBlock lb = getFollowingBlock();\n    block.setCurrentBlock(lb.getBlock());\n    block.setNumBytes(0);\n    bytesSent = 0;\n    accessToken = lb.getBlockToken();\n\n    DatanodeInfo[] nodes = lb.getLocations();\n    StorageType[] storageTypes = lb.getStorageTypes();\n    String[] storageIDs = lb.getStorageIDs();\n    // Connect to the DataNode. If fail the internal error state will be set.\n    success = createBlockOutputStream(nodes, storageTypes, storageIDs, 0L,\n        false);\n\n    if (!success) {\n      block.setCurrentBlock(null);\n      final DatanodeInfo badNode = nodes[getErrorState().getBadNodeIndex()];\n      LOG.warn(\"Excluding datanode \" + badNode);\n      excludedNodes.put(badNode, badNode);\n      throw new IOException(\"Unable to create new block.\" + this);\n    }\n    setPipeline(lb);\n  }\n\n  @VisibleForTesting\n  LocatedBlock peekFollowingBlock() {\n    return coordinator.getFollowingBlocks().peek(index);\n  }\n\n  @Override\n  protected boolean setupPipelineInternal(DatanodeInfo[] nodes,\n      StorageType[] nodeStorageTypes, String[] nodeStorageIDs)\n      throws IOException {\n    boolean success = false;\n    while (!success && !streamerClosed() && dfsClient.clientRunning) {\n      if (!handleRestartingDatanode()) {\n        return false;\n      }","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java#L95-L131","documentation":"StripedDataStreamer, the writer-side streamer for erasure-coded files, failed to set up its block pipeline: createBlockOutputStream returned false for the LocatedBlock's datanodes. It identifies the bad node, adds it to excludedNodes, resets the current block, and throws this IOException so DFSStripedOutputStream can request a fresh block.","triggerScenarios":"Writing an EC file when pipeline creation fails: the chosen datanode is down or unreachable, already excluded, at its xceiver limit (dfs.datanode.max.transfer.threads), out of disk, or the cluster has fewer healthy datanodes than the EC policy requires.","commonSituations":"Test clusters smaller than dataUnits+parityUnits (e.g. RS-3-2 needs 5 nodes but only 4 are up); datanode disk full; xceiver thread exhaustion under heavy EC writes; repeated failures accumulate exclusions until no candidates remain.","solutions":["Ensure the cluster has at least (numDataUnits + numParityUnits) healthy writable datanodes for the EC policy in use (hdfs dfsadmin -report)","Inspect DataNode logs for the refused pipeline: xceiver limit, disk space, Kerberos/SASL handshake failures","Raise dfs.datanode.max.transfer.threads if xceiver exhaustion shows in the logs","Retry the write: exclusions are per-stream, so a new stream can pick recovered or re-replicated nodes"],"exampleFix":"// before: one-shot EC write\ntry (FSDataOutputStream out = fs.create(ecPath)) {\n  out.write(data);\n}\n\n// after: retry with backoff to survive transient pipeline failures\nfor (int i = 0; ; i++) {\n  try (FSDataOutputStream out = fs.create(ecPath, true)) {\n    out.write(data);\n    break;\n  } catch (IOException e) {\n    if (i == maxRetries - 1) throw e;\n    Thread.sleep(1000L << i);\n  }\n}","handlingStrategy":"retry","validationCode":"// Verify enough live datanodes for the EC policy before writing\nErasureCodingPolicy p = dfs.getErasureCodingPolicy(ecPath);\nDatanodeInfo[] live = dfs.getDataNodeStats(HdfsConstants.DatanodeReportTypes.LIVE);\nint needed = p.getNumDataUnits() + p.getNumParityUnits();\nif (live.length < needed) {\n  throw new IllegalStateException(\n      \"need \" + needed + \" live datanodes, have \" + live.length);\n}","typeGuard":null,"tryCatchPattern":"try {\n  writeEc(path, data);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Unable to create new block\")) {\n    // excluded-node set is per-stream; a new stream retries fresh pipelines\n    writeEc(path, data);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Size clusters to at least dataUnits+parityUnits healthy datanodes for every EC policy in use","Watch DataNode xceiver counts and disk space; raise dfs.datanode.max.transfer.threads before it saturates","Add application-level retry around EC creates; per-stream exclusions reset on a new stream"],"tags":["hdfs","erasure-coding","write","pipeline","datanode"],"backgroundTag":"pipeline-setup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}