apache/hadoop · error · IOException

Unable to create new block.{}

Error message

Unable to create new block.{}

What it means

StripedDataStreamer, the writer-side streamer for erasure-coded files, failed to set up its block pipeline: createBlockOutputStream returned false for the LocatedBlock's datanodes. It identifies the bad node, adds it to excludedNodes, resets the current block, and throws this IOException so DFSStripedOutputStream can request a fresh block.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java:113

    LocatedBlock lb = getFollowingBlock();
    block.setCurrentBlock(lb.getBlock());
    block.setNumBytes(0);
    bytesSent = 0;
    accessToken = lb.getBlockToken();

    DatanodeInfo[] nodes = lb.getLocations();
    StorageType[] storageTypes = lb.getStorageTypes();
    String[] storageIDs = lb.getStorageIDs();
    // Connect to the DataNode. If fail the internal error state will be set.
    success = createBlockOutputStream(nodes, storageTypes, storageIDs, 0L,
        false);

    if (!success) {
      block.setCurrentBlock(null);
      final DatanodeInfo badNode = nodes[getErrorState().getBadNodeIndex()];
      LOG.warn("Excluding datanode " + badNode);
      excludedNodes.put(badNode, badNode);
      throw new IOException("Unable to create new block." + this);
    }
    setPipeline(lb);
  }

  @VisibleForTesting
  LocatedBlock peekFollowingBlock() {
    return coordinator.getFollowingBlocks().peek(index);
  }

  @Override
  protected boolean setupPipelineInternal(DatanodeInfo[] nodes,
      StorageType[] nodeStorageTypes, String[] nodeStorageIDs)
      throws IOException {
    boolean success = false;
    while (!success && !streamerClosed() && dfsClient.clientRunning) {
      if (!handleRestartingDatanode()) {
        return false;
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure the cluster has at least (numDataUnits + numParityUnits) healthy writable datanodes for the EC policy in use (hdfs dfsadmin -report)
2. Inspect DataNode logs for the refused pipeline: xceiver limit, disk space, Kerberos/SASL handshake failures
3. Raise dfs.datanode.max.transfer.threads if xceiver exhaustion shows in the logs
4. Retry the write: exclusions are per-stream, so a new stream can pick recovered or re-replicated nodes

Example fix

// before: one-shot EC write
try (FSDataOutputStream out = fs.create(ecPath)) {
  out.write(data);
}

// after: retry with backoff to survive transient pipeline failures
for (int i = 0; ; i++) {
  try (FSDataOutputStream out = fs.create(ecPath, true)) {
    out.write(data);
    break;
  } catch (IOException e) {
    if (i == maxRetries - 1) throw e;
    Thread.sleep(1000L << i);
  }
}

// Verify enough live datanodes for the EC policy before writing
ErasureCodingPolicy p = dfs.getErasureCodingPolicy(ecPath);
DatanodeInfo[] live = dfs.getDataNodeStats(HdfsConstants.DatanodeReportTypes.LIVE);
int needed = p.getNumDataUnits() + p.getNumParityUnits();
if (live.length < needed) {
  throw new IllegalStateException(
      "need " + needed + " live datanodes, have " + live.length);
}

try {
  writeEc(path, data);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to create new block")) {
    // excluded-node set is per-stream; a new stream retries fresh pipelines
    writeEc(path, data);
  } else {
    throw e;
  }
}

When it happens

Trigger: Writing an EC file when pipeline creation fails: the chosen datanode is down or unreachable, already excluded, at its xceiver limit (dfs.datanode.max.transfer.threads), out of disk, or the cluster has fewer healthy datanodes than the EC policy requires.

Common situations: Test clusters smaller than dataUnits+parityUnits (e.g. RS-3-2 needs 5 nodes but only 4 are up); datanode disk full; xceiver thread exhaustion under heavy EC writes; repeated failures accumulate exclusions until no candidates remain.

Related errors

• Unknown BlockChecksumType: " + groupChecksumType
• {} has no enough internal blocks(current: {}), unable to sta
• rur.getNumBytes() < newlength = {newlength}, rur={rur}
• Invalid values: dfs.bytes-per-checksum (={}) must divide cel
• Failed: the number of failed blocks = {} > the number of fai

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/baea183c7ded0068. Report an issue: GitHub.