apache/hadoop · error · IOException

All datanodes failed: block={}, datanodeids={}

Error message

All datanodes failed: block={}, datanodeids={}

What it means

Error "All datanodes failed: block={}, datanodeids={}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:170

            } else {
              LOG.debug("Block recovery: Ignored replica with invalid "
                  + "generation stamp or length: {} from DataNode: {}", info, id);
            }
          }
        } catch (RecoveryInProgressException ripE) {
          InterDatanodeProtocol.LOG.warn(
              "Recovery for replica {} on data-node {} is already in progress. " +
                  "Recovery id = {} is aborted.", block, id, rBlock.getNewGenerationStamp(), ripE);
          return;
        } catch (IOException e) {
          ++errorCount;
          InterDatanodeProtocol.LOG.warn("Failed to recover block (block={}, datanode={})",
              block, id, e);
        }
      }

      if (errorCount == locs.length) {
        throw new IOException("All datanodes failed: block=" + block
            + ", datanodeids=" + Arrays.asList(locs));
      }

      // None of the replicas reported by DataNodes has the required original
      // state, report the error.
      if (candidateReplicaCnt > 0 && syncList.isEmpty()) {
        throw new IOException("Found " + candidateReplicaCnt +
            " replica(s) for block " + block + " but none is in " +
            ReplicaState.RWR.name() + " or better state. datanodeids=" +
            Arrays.asList(locs));
      }

      syncBlock(syncList);
    }

    /** Block synchronization. */
    void syncBlock(List<BlockRecord> syncList) throws IOException {
      DatanodeProtocolClientSideTranslatorPB nn =

View on GitHub (pinned to 2add963021)

Solutions

1. Check health and reachability of every DataNode holding a replica of the block; restart or decommission failed nodes and re-run block recovery.
2. If all replicas are lost, restore the file from backup or accept the loss; recovery needs at least one reachable replica.

When it happens

Trigger: Block recovery (lease recovery) fails because every DataNode listed as holding the block is dead or unreachable.

Common situations: Block recovery failed because every DataNode holding the block was unreachable or errored. Verify the listed DataNodes are live and network-reachable.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fbdc4ea8863f92ab. Report an issue: GitHub.