apache/hadoop · critical · IOException

File %s could only be written to %d of the %d %s. There are

Error message

File %s could only be written to %d of the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this operation.

What it means

Thrown by BlockManager.chooseTarget4NewBlock (via verifyBlockPlacement/chooseTarget) for a regular replicated (CONTIGUOUS) file when the placement policy could not pick enough target datanodes: fewer than minReplication (typically 1, or the file's minimum) replicas could be placed. The message reports how many targets were chosen, the required minimum, live datanode count (topology leaves), and how many nodes the client excluded. It is the classic 'cannot write: not enough datanodes' error at block allocation time.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:2492

      final byte storagePolicyID,
      final BlockType blockType,
      final ErasureCodingPolicy ecPolicy,
      final EnumSet<AddBlockFlag> flags) throws IOException {
    List<DatanodeDescriptor> favoredDatanodeDescriptors = 
        getDatanodeDescriptors(favoredNodes);
    final BlockStoragePolicy storagePolicy =
        storagePolicySuite.getPolicy(storagePolicyID);
    final BlockPlacementPolicy blockplacement =
        placementPolicies.getPolicy(blockType);
    final DatanodeStorageInfo[] targets = blockplacement.chooseTarget(src,
        numOfReplicas, client, excludedNodes, blocksize, 
        favoredDatanodeDescriptors, storagePolicy, flags);

    final String errorMessage = "File %s could only be written to %d of " +
        "the %d %s. There are %d datanode(s) running and %s "
        + "node(s) are excluded in this operation.";
    if (blockType == BlockType.CONTIGUOUS && targets.length < minReplication) {
      throw new IOException(String.format(errorMessage, src,
          targets.length, minReplication, "minReplication nodes",
          getDatanodeManager().getNetworkTopology().getNumOfLeaves(),
          (excludedNodes == null? "no": excludedNodes.size())));
    } else if (blockType == BlockType.STRIPED &&
        targets.length < ecPolicy.getNumDataUnits()) {
      throw new IOException(
          String.format(errorMessage, src, targets.length,
              ecPolicy.getNumDataUnits(),
              String.format("required nodes for %s", ecPolicy.getName()),
              getDatanodeManager().getNetworkTopology().getNumOfLeaves(),
              (excludedNodes == null ? "no" : excludedNodes.size())));
    }
    return targets;
  }

  /**
   * Get list of datanode descriptors for given list of nodes. Nodes are
   * hostaddress:port or just hostaddress.

View on GitHub (pinned to 2add963021)

Solutions

1. Check `hdfs dfsadmin -report`: live datanodes must be >= desired replication; restart dead DNs or add nodes
2. For dev/test single node: set replication to 1 (fs.setReplication or -Ddfs.replication=1 at create time)
3. Free datanode disk space / raise volume failure tolerance so DNs become choosable again
4. If caused by excluded nodes after transient failures, retry the write with a fresh exclusion set; fix the underlying DN connectivity

Example fix

// before: 1-node cluster, default replication 3
FSDataOutputStream out = fs.create(path); // addBlock -> could only be written to 0 of 1 minReplication

// after: request replication that the cluster can satisfy
FSDataOutputStream out = fs.create(path, (short) 1, true, 4096, (short) 1);

// Pre-flight: can the cluster satisfy replication at create time?
DistributedFileSystem dfs = (DistributedFileSystem) fs;
int live = dfs.getDataNodeStats(DatanodeReportTypes.LIVE).length;
short rep = (short) Math.min(requestedReplication, Math.max(1, live));
if (live < 1) throw new IOException("No live datanodes; cannot write");
try (FSDataOutputStream out = dfs.create(path, rep, true, 1 << 16, rep)) { ... }

try {
  out = fs.create(path, rep);
} catch (IOException e) {
  if (e.getMessage().contains("could only be written to")) {
    // inspect message for live/excluded counts; degrade replication to live nodes
    out = fs.create(path, (short) Math.max(1, liveCount));
  } else { throw e; }
}

When it happens

Trigger: Client calls create/append -> addBlock while total live datanodes < needed replicas, or excludedNodes (client-supplied + tried-and-failed targets, favored-nodes constraints) plus decommissioning/stale/capacity-full nodes leave fewer than minReplication choosable targets.

Common situations: Single-node test cluster with replication=3; nodes down/decommissioning; disks full so DNs are not chosen; client excludes failed DNs across retries (ReplicaWriter keepalive exhaustion); rack policy needing 2 racks with only 1 available.

Related errors

• Balancer without BlockPlacementPolicyDefault
• Unexpected configuration parameters: dfs.namenode.replicatio
• Unexpected configuration parameters: dfs.replication.max = {
• Unexpected configuration parameters: dfs.namenode.replicatio
• Unexpected configuration parameters: dfs.namenode.maintenanc

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/08dbdb789d5cb6f2. Report an issue: GitHub.