apache/hadoop · critical · IOException

Failed: the number of failed blocks = {} > the number of fai

Error message

Failed: the number of failed blocks = {} > the number of failed blocks tolerated = {}

What it means

DFSStripedOutputStream tracks one DataStreamer per data/parity block while writing an erasure-coded file. checkStreamers() totals known-failed streamers plus newly failed ones; if that count exceeds failedBlocksTolerated — by default the policy's parity count (dfs.client.ec.write.failed.blocks.tolerated, default -1 meaning parity units, capped at parity) — the streamer set can no longer produce a recoverable stripe. The client closes every streamer and throws IOException('Failed: the number of failed blocks = N > the number of failed blocks tolerated = M').

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java:420

   */
  private Set<StripedDataStreamer> checkStreamers() throws IOException {
    Set<StripedDataStreamer> newFailed = new HashSet<>();
    for(StripedDataStreamer s : streamers) {
      if (!s.isHealthy() && !failedStreamers.contains(s)) {
        newFailed.add(s);
      }
    }

    final int failCount = failedStreamers.size() + newFailed.size();
    if (LOG.isDebugEnabled()) {
      LOG.debug("checkStreamers: {}", streamers);
      LOG.debug("healthy streamer count={}", (numAllBlocks - failCount));
      LOG.debug("original failed streamers: {}", failedStreamers);
      LOG.debug("newly failed streamers: {}", newFailed);
    }
    if (failCount > failedBlocksTolerated) {
      closeAllStreamers();
      throw new IOException("Failed: the number of failed blocks = "
          + failCount + " > the number of failed blocks tolerated = "
          + failedBlocksTolerated);
    }
    return newFailed;
  }

  private void closeAllStreamers() {
    // The write has failed, Close all the streamers.
    for (StripedDataStreamer streamer : streamers) {
      streamer.close(true);
    }
  }

  private void handleCurrentStreamerFailure(String err, Exception e)
      throws IOException {
    currentPacket = null;
    handleStreamerFailure(err, e, getCurrentStreamer());
  }

View on GitHub (pinned to 2add963021)

Solutions

Identify the failed streamers from the preceding DEBUG logs (failedStreamers / newly failed) and fix the corresponding datanodes or network before retrying.
Verify with hdfs dfsadmin -report that at least dataBlocks + parity datanodes are healthy and have write capacity.
Delete the partially written file and rerun the write job once the nodes are healthy — the abandoned file cannot be completed.
If the cluster cannot sustain the policy, write that path with replication (setStoragePolicy REPLICATED / delete EC policy dir flag) or a smaller policy like RS-3-2-1024k.

Example fix

# before: EC write on a cluster with too few healthy datanodes
hdfs dfs -Ddfs.replication=... -put big.file /ec-rs-6-3/big.file  # policy on dir is RS-6-3-1024k

# after: give the path a policy the cluster can sustain, or replicate
hdfs ec -setPolicy -policy RS-3-2-1024k -path /small-cluster
# or
hdfs storagepolicies -setStoragePolicy -path /big -policy REPLICATED

Defensive patterns

Strategy: retry

Validate before calling

DistributedFileSystem dfs = (DistributedFileSystem) fs;
DatanodeInfo[] live = dfs.getDataNodeStats(DatanodeReportTypes.LIVE);
ErasureCodingPolicy ecPolicy = dfs.getErasureCodingPolicy(ecPath);
if (ecPolicy != null
    && live.length < ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits()) {
  // not enough healthy datanodes: EC writes will likely fail
  LOG.warn("Only {} live datanodes for policy {} needing {}",
      live.length, ecPolicy.getName(),
      ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits());
}

Try / catch

try {
  writeFileWithRetry(fs, src, data); // fails after >parity streamer failures
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("failed blocks")) {
    // cluster cannot sustain the EC write: fix nodes, delete partial file,
    // optionally retry with a smaller policy or REPLICATED storage
  }
  throw e;
}

Prevention

Monitor datanode health and free capacity; EC writes need data+parity healthy writers.
For small or churn-prone clusters, use lower-parity policies (RS-3-2-1024k) or replication for write-heavy paths.
Delete partial files after this failure — they cannot be completed (fsck reports them under-replicated/corrupt).
dfs.client.ec.write.failed.blocks.tolerated exists but is capped at parity count — it cannot exceed what the policy can repair.

When it happens

Trigger: Writing an EC file while more block writers fail than the policy's parity count: multiple datanodes refusing/failing pipelines (dead nodes, full disks, network partitions) during write, flush, or close. checkStreamers() runs on the error path after stripe write failures, so the exception surfaces from the next write()/hflush()/close().

Common situations: Clusters with fewer healthy datanodes than data+parity units; rolling restarts or dying disks during long EC writes; small test clusters using RS-6-3-1024k with fewer than 9 writable nodes; repeated streamer failures on one bad rack.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fe367eb2a82aa75b. Report an issue: GitHub.