{"record":{"id":"e485bdccf419a2a4","repo":"apache/hadoop","slug":"cannot-recover-the-following-datanodes-failed","errorCode":null,"errorMessage":"Cannot recover {}, the following datanodes failed: {}","messagePattern":"Cannot recover (.+?), the following datanodes failed: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java","lineNumber":301,"sourceCode":"          newBlock.getNumBytes(), participatingList);\n\n      List<DatanodeID> failedList = new ArrayList<>();\n      final List<BlockRecord> successList = new ArrayList<>();\n      for (BlockRecord r : participatingList) {\n        try {\n          r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,\n              newBlock.getNumBytes());\n          successList.add(r);\n        } catch (IOException e) {\n          InterDatanodeProtocol.LOG.warn(\"Failed to updateBlock (newblock=\"\n              + newBlock + \", datanode=\" + r.id + \")\", e);\n          failedList.add(r.id);\n        }\n      }\n\n      // Abort if all failed.\n      if (successList.isEmpty()) {\n        throw new IOException(\"Cannot recover \" + block\n            + \", the following datanodes failed: \" + failedList);\n      }\n\n      // Notify the name-node about successfully recovered replicas.\n      final DatanodeID[] datanodes = new DatanodeID[successList.size()];\n      final String[] storages = new String[datanodes.length];\n      for (int i = 0; i < datanodes.length; i++) {\n        final BlockRecord r = successList.get(i);\n        datanodes[i] = r.id;\n        storages[i] = r.storageID;\n      }\n\n      LOG.debug(\"Datanode triggering commitBlockSynchronization, block={}, newGs={}, \" +\n          \"newLength={}\", block, newBlock.getGenerationStamp(), newBlock.getNumBytes());\n\n      nn.commitBlockSynchronization(block,\n          newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,\n          datanodes, storages);","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L283-L319","documentation":"Error \"Cannot recover {}, the following datanodes failed: {}\" thrown in apache/hadoop.","triggerScenarios":"Block recovery for a file fails because one or more DataNodes participating in the recovery did not respond successfully.","commonSituations":"Block recovery could not complete because some participating DataNodes failed. Recoverable: fix or decommission the failed nodes and rerun recovery.","solutions":["Restore connectivity or restart the failed DataNodes listed in the message, then re-run block recovery (it is retried automatically by lease recovery).","If some DataNodes are permanently gone, force recovery with the remaining replicas or restore the file from backup."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}