apache/hadoop · error · IOException

Cannot recover {}, the following datanodes failed: {}

Error message

Cannot recover {}, the following datanodes failed: {}

What it means

Error "Cannot recover {}, the following datanodes failed: {}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:301

          newBlock.getNumBytes(), participatingList);

      List<DatanodeID> failedList = new ArrayList<>();
      final List<BlockRecord> successList = new ArrayList<>();
      for (BlockRecord r : participatingList) {
        try {
          r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,
              newBlock.getNumBytes());
          successList.add(r);
        } catch (IOException e) {
          InterDatanodeProtocol.LOG.warn("Failed to updateBlock (newblock="
              + newBlock + ", datanode=" + r.id + ")", e);
          failedList.add(r.id);
        }
      }

      // Abort if all failed.
      if (successList.isEmpty()) {
        throw new IOException("Cannot recover " + block
            + ", the following datanodes failed: " + failedList);
      }

      // Notify the name-node about successfully recovered replicas.
      final DatanodeID[] datanodes = new DatanodeID[successList.size()];
      final String[] storages = new String[datanodes.length];
      for (int i = 0; i < datanodes.length; i++) {
        final BlockRecord r = successList.get(i);
        datanodes[i] = r.id;
        storages[i] = r.storageID;
      }

      LOG.debug("Datanode triggering commitBlockSynchronization, block={}, newGs={}, " +
          "newLength={}", block, newBlock.getGenerationStamp(), newBlock.getNumBytes());

      nn.commitBlockSynchronization(block,
          newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
          datanodes, storages);

View on GitHub (pinned to 2add963021)

Solutions

1. Restore connectivity or restart the failed DataNodes listed in the message, then re-run block recovery (it is retried automatically by lease recovery).
2. If some DataNodes are permanently gone, force recovery with the remaining replicas or restore the file from backup.

When it happens

Trigger: Block recovery for a file fails because one or more DataNodes participating in the recovery did not respond successfully.

Common situations: Block recovery could not complete because some participating DataNodes failed. Recoverable: fix or decommission the failed nodes and rerun recovery.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e485bdccf419a2a4. Report an issue: GitHub.