{"record":{"id":"162e34f61a5890f8","repo":"apache/hadoop","slug":"data-streamers-failed-while-creating-new-block-str","errorCode":null,"errorMessage":"Data streamers failed while creating new block streams: {}. There are not enough healthy streamers.","messagePattern":"Data streamers failed while creating new block streams: (.+?)\\. There are not enough healthy streamers\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java","lineNumber":706,"sourceCode":"      failedStreamers.addAll(newFailed);\n      coordinator.clearFailureStates();\n      corruptBlockCountMap.put(blockGroupIndex, failedStreamers.size());\n\n      // mark all the healthy streamers as external error\n      Set<StripedDataStreamer> healthySet = markExternalErrorOnStreamers();\n\n      // we have newly failed streamers, update block for pipeline\n      final ExtendedBlock newBG = updateBlockForPipeline(healthySet);\n\n      // wait till all the healthy streamers to\n      // 1) get the updated block info\n      // 2) create new block outputstream\n      newFailed = waitCreatingStreamers(healthySet);\n      if (newFailed.size() + failedStreamers.size() >\n          failedBlocksTolerated) {\n        // The write has failed, Close all the streamers.\n        closeAllStreamers();\n        throw new IOException(\n            \"Data streamers failed while creating new block streams: \"\n                + newFailed + \". There are not enough healthy streamers.\");\n      }\n      for (StripedDataStreamer failedStreamer : newFailed) {\n        assert !failedStreamer.isHealthy();\n      }\n\n      // TODO we can also succeed if all the failed streamers have not taken\n      // the updated block\n      if (newFailed.size() == 0) {\n        // reset external error state of all the streamers\n        for (StripedDataStreamer streamer : healthySet) {\n          assert streamer.isHealthy();\n          streamer.getErrorState().reset();\n        }\n        updatePipeline(newBG);\n      }\n      for (int i = 0; i < numAllBlocks; i++) {","sourceCodeStart":688,"sourceCodeEnd":724,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java#L688-L724","documentation":"This is the recovery-time sibling of the failed-blocks check: after a streamer failure, DFSStripedOutputStream rebuilds pipelines — it gets a new block generation for the pipeline (updateBlockForPipeline), has the healthy streamers create new block output streams, and re-checks them (waitCreatingStreamers). If streamers that were healthy at the start of recovery fail during this transition and failedStreamers + newFailed exceeds failedBlocksTolerated (parity count by default), all streamers are closed and IOException('Data streamers failed while creating new block streams: [...]. There are not enough healthy streamers.') is thrown.","triggerScenarios":"A second wave of datanode/network failures landing exactly while pipelines are being re-established after an initial failure — e.g., one DN dies, recovery starts, and another DN or link fails while healthy streamers reconnect, pushing cumulative failures past what parity can repair.","commonSituations":"Unstable clusters during writes (rolling upgrades, network churn, batch disk failures); low-parity policies (XOR-2-1-1024k tolerates a single failure) where one extra hiccup during recovery aborts the write; saturated clusters where re-pipeline requests time out and streamers get marked failed.","solutions":["Fix the underlying instability first — check the datanodes named in the exception message and their logs at that timestamp.","Confirm enough healthy datanodes exist for data+parity units, then delete the partial file and rerun the write.","Prefer a higher-parity policy (RS-6-3 over XOR-2-1/RS-3-2) on clusters with expected churn.","If failures concentrate on one host or rack, exclude/decommission it before retrying."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  copyLargeFileToEcPath(src, ecDest);\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"not enough healthy streamers\")) {\n    // recovery-time failures exceeded parity: stabilize cluster, cleanup,\n    // then retry the job\n    cleanupPartial(ecDest);\n    retryAfterClusterStabilizes();\n  } else {\n    throw e;\n  }\n}","preventionTips":["Avoid writes during rolling upgrades/restarts on EC-coded paths.","Prefer higher-parity policies (RS-6-3-1024k) where single failures during recovery must not kill the write.","Make bulk write jobs idempotent and restartable so a re-run is cheap after cluster stabilization."],"tags":["hdfs","erasure-coding","write","pipeline-recovery","datanode-failure"],"backgroundTag":"datanode-write-pipeline-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}