apache/hadoop · critical · IOException

Data streamers failed while creating new block streams: {}.

Error message

Data streamers failed while creating new block streams: {}. There are not enough healthy streamers.

What it means

This is the recovery-time sibling of the failed-blocks check: after a streamer failure, DFSStripedOutputStream rebuilds pipelines — it gets a new block generation for the pipeline (updateBlockForPipeline), has the healthy streamers create new block output streams, and re-checks them (waitCreatingStreamers). If streamers that were healthy at the start of recovery fail during this transition and failedStreamers + newFailed exceeds failedBlocksTolerated (parity count by default), all streamers are closed and IOException('Data streamers failed while creating new block streams: [...]. There are not enough healthy streamers.') is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java:706

      failedStreamers.addAll(newFailed);
      coordinator.clearFailureStates();
      corruptBlockCountMap.put(blockGroupIndex, failedStreamers.size());

      // mark all the healthy streamers as external error
      Set<StripedDataStreamer> healthySet = markExternalErrorOnStreamers();

      // we have newly failed streamers, update block for pipeline
      final ExtendedBlock newBG = updateBlockForPipeline(healthySet);

      // wait till all the healthy streamers to
      // 1) get the updated block info
      // 2) create new block outputstream
      newFailed = waitCreatingStreamers(healthySet);
      if (newFailed.size() + failedStreamers.size() >
          failedBlocksTolerated) {
        // The write has failed, Close all the streamers.
        closeAllStreamers();
        throw new IOException(
            "Data streamers failed while creating new block streams: "
                + newFailed + ". There are not enough healthy streamers.");
      }
      for (StripedDataStreamer failedStreamer : newFailed) {
        assert !failedStreamer.isHealthy();
      }

      // TODO we can also succeed if all the failed streamers have not taken
      // the updated block
      if (newFailed.size() == 0) {
        // reset external error state of all the streamers
        for (StripedDataStreamer streamer : healthySet) {
          assert streamer.isHealthy();
          streamer.getErrorState().reset();
        }
        updatePipeline(newBG);
      }
      for (int i = 0; i < numAllBlocks; i++) {

View on GitHub (pinned to 2add963021)

Solutions

Fix the underlying instability first — check the datanodes named in the exception message and their logs at that timestamp.
Confirm enough healthy datanodes exist for data+parity units, then delete the partial file and rerun the write.
Prefer a higher-parity policy (RS-6-3 over XOR-2-1/RS-3-2) on clusters with expected churn.
If failures concentrate on one host or rack, exclude/decommission it before retrying.

Defensive patterns

Strategy: retry

Try / catch

try {
  copyLargeFileToEcPath(src, ecDest);
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("not enough healthy streamers")) {
    // recovery-time failures exceeded parity: stabilize cluster, cleanup,
    // then retry the job
    cleanupPartial(ecDest);
    retryAfterClusterStabilizes();
  } else {
    throw e;
  }
}

Prevention

Avoid writes during rolling upgrades/restarts on EC-coded paths.
Prefer higher-parity policies (RS-6-3-1024k) where single failures during recovery must not kill the write.
Make bulk write jobs idempotent and restartable so a re-run is cheap after cluster stabilization.

When it happens

Trigger: A second wave of datanode/network failures landing exactly while pipelines are being re-established after an initial failure — e.g., one DN dies, recovery starts, and another DN or link fails while healthy streamers reconnect, pushing cumulative failures past what parity can repair.

Common situations: Unstable clusters during writes (rolling upgrades, network churn, batch disk failures); low-parity policies (XOR-2-1-1024k tolerates a single failure) where one extra hiccup during recovery aborts the write; saturated clusters where re-pipeline requests time out and streamers get marked failed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/162e34f61a5890f8. Report an issue: GitHub.