{"record":{"id":"fe367eb2a82aa75b","repo":"apache/hadoop","slug":"failed-the-number-of-failed-blocks-the-num","errorCode":null,"errorMessage":"Failed: the number of failed blocks = {} > the number of failed blocks tolerated = {}","messagePattern":"Failed: the number of failed blocks = (.+?) > the number of failed blocks tolerated = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java","lineNumber":420,"sourceCode":"   */\n  private Set<StripedDataStreamer> checkStreamers() throws IOException {\n    Set<StripedDataStreamer> newFailed = new HashSet<>();\n    for(StripedDataStreamer s : streamers) {\n      if (!s.isHealthy() && !failedStreamers.contains(s)) {\n        newFailed.add(s);\n      }\n    }\n\n    final int failCount = failedStreamers.size() + newFailed.size();\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"checkStreamers: {}\", streamers);\n      LOG.debug(\"healthy streamer count={}\", (numAllBlocks - failCount));\n      LOG.debug(\"original failed streamers: {}\", failedStreamers);\n      LOG.debug(\"newly failed streamers: {}\", newFailed);\n    }\n    if (failCount > failedBlocksTolerated) {\n      closeAllStreamers();\n      throw new IOException(\"Failed: the number of failed blocks = \"\n          + failCount + \" > the number of failed blocks tolerated = \"\n          + failedBlocksTolerated);\n    }\n    return newFailed;\n  }\n\n  private void closeAllStreamers() {\n    // The write has failed, Close all the streamers.\n    for (StripedDataStreamer streamer : streamers) {\n      streamer.close(true);\n    }\n  }\n\n  private void handleCurrentStreamerFailure(String err, Exception e)\n      throws IOException {\n    currentPacket = null;\n    handleStreamerFailure(err, e, getCurrentStreamer());\n  }","sourceCodeStart":402,"sourceCodeEnd":438,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java#L402-L438","documentation":"DFSStripedOutputStream tracks one DataStreamer per data/parity block while writing an erasure-coded file. checkStreamers() totals known-failed streamers plus newly failed ones; if that count exceeds failedBlocksTolerated — by default the policy's parity count (dfs.client.ec.write.failed.blocks.tolerated, default -1 meaning parity units, capped at parity) — the streamer set can no longer produce a recoverable stripe. The client closes every streamer and throws IOException('Failed: the number of failed blocks = N > the number of failed blocks tolerated = M').","triggerScenarios":"Writing an EC file while more block writers fail than the policy's parity count: multiple datanodes refusing/failing pipelines (dead nodes, full disks, network partitions) during write, flush, or close. checkStreamers() runs on the error path after stripe write failures, so the exception surfaces from the next write()/hflush()/close().","commonSituations":"Clusters with fewer healthy datanodes than data+parity units; rolling restarts or dying disks during long EC writes; small test clusters using RS-6-3-1024k with fewer than 9 writable nodes; repeated streamer failures on one bad rack.","solutions":["Identify the failed streamers from the preceding DEBUG logs (failedStreamers / newly failed) and fix the corresponding datanodes or network before retrying.","Verify with hdfs dfsadmin -report that at least dataBlocks + parity datanodes are healthy and have write capacity.","Delete the partially written file and rerun the write job once the nodes are healthy — the abandoned file cannot be completed.","If the cluster cannot sustain the policy, write that path with replication (setStoragePolicy REPLICATED / delete EC policy dir flag) or a smaller policy like RS-3-2-1024k."],"exampleFix":"# before: EC write on a cluster with too few healthy datanodes\nhdfs dfs -Ddfs.replication=... -put big.file /ec-rs-6-3/big.file  # policy on dir is RS-6-3-1024k\n\n# after: give the path a policy the cluster can sustain, or replicate\nhdfs ec -setPolicy -policy RS-3-2-1024k -path /small-cluster\n# or\nhdfs storagepolicies -setStoragePolicy -path /big -policy REPLICATED","handlingStrategy":"retry","validationCode":"DistributedFileSystem dfs = (DistributedFileSystem) fs;\nDatanodeInfo[] live = dfs.getDataNodeStats(DatanodeReportTypes.LIVE);\nErasureCodingPolicy ecPolicy = dfs.getErasureCodingPolicy(ecPath);\nif (ecPolicy != null\n    && live.length < ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits()) {\n  // not enough healthy datanodes: EC writes will likely fail\n  LOG.warn(\"Only {} live datanodes for policy {} needing {}\",\n      live.length, ecPolicy.getName(),\n      ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits());\n}","typeGuard":null,"tryCatchPattern":"try {\n  writeFileWithRetry(fs, src, data); // fails after >parity streamer failures\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"failed blocks\")) {\n    // cluster cannot sustain the EC write: fix nodes, delete partial file,\n    // optionally retry with a smaller policy or REPLICATED storage\n  }\n  throw e;\n}","preventionTips":["Monitor datanode health and free capacity; EC writes need data+parity healthy writers.","For small or churn-prone clusters, use lower-parity policies (RS-3-2-1024k) or replication for write-heavy paths.","Delete partial files after this failure — they cannot be completed (fsck reports them under-replicated/corrupt).","dfs.client.ec.write.failed.blocks.tolerated exists but is capped at parity count — it cannot exceed what the policy can repair."],"tags":["hdfs","erasure-coding","write","pipeline-failure","datanode-failure"],"backgroundTag":"datanode-write-pipeline-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}