{"record":{"id":"fbdc4ea8863f92ab","repo":"apache/hadoop","slug":"all-datanodes-failed-block-datanodeids","errorCode":null,"errorMessage":"All datanodes failed: block={}, datanodeids={}","messagePattern":"All datanodes failed: block=(.+?), datanodeids=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java","lineNumber":170,"sourceCode":"            } else {\n              LOG.debug(\"Block recovery: Ignored replica with invalid \"\n                  + \"generation stamp or length: {} from DataNode: {}\", info, id);\n            }\n          }\n        } catch (RecoveryInProgressException ripE) {\n          InterDatanodeProtocol.LOG.warn(\n              \"Recovery for replica {} on data-node {} is already in progress. \" +\n                  \"Recovery id = {} is aborted.\", block, id, rBlock.getNewGenerationStamp(), ripE);\n          return;\n        } catch (IOException e) {\n          ++errorCount;\n          InterDatanodeProtocol.LOG.warn(\"Failed to recover block (block={}, datanode={})\",\n              block, id, e);\n        }\n      }\n\n      if (errorCount == locs.length) {\n        throw new IOException(\"All datanodes failed: block=\" + block\n            + \", datanodeids=\" + Arrays.asList(locs));\n      }\n\n      // None of the replicas reported by DataNodes has the required original\n      // state, report the error.\n      if (candidateReplicaCnt > 0 && syncList.isEmpty()) {\n        throw new IOException(\"Found \" + candidateReplicaCnt +\n            \" replica(s) for block \" + block + \" but none is in \" +\n            ReplicaState.RWR.name() + \" or better state. datanodeids=\" +\n            Arrays.asList(locs));\n      }\n\n      syncBlock(syncList);\n    }\n\n    /** Block synchronization. */\n    void syncBlock(List<BlockRecord> syncList) throws IOException {\n      DatanodeProtocolClientSideTranslatorPB nn =","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L152-L188","documentation":"Error \"All datanodes failed: block={}, datanodeids={}\" thrown in apache/hadoop.","triggerScenarios":"Block recovery (lease recovery) fails because every DataNode listed as holding the block is dead or unreachable.","commonSituations":"Block recovery failed because every DataNode holding the block was unreachable or errored. Verify the listed DataNodes are live and network-reachable.","solutions":["Check health and reachability of every DataNode holding a replica of the block; restart or decommission failed nodes and re-run block recovery.","If all replicas are lost, restore the file from backup or accept the loss; recovery needs at least one reachable replica."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}