{"record":{"id":"999633b2875f4c33","repo":"apache/hadoop","slug":"has-no-enough-internal-blocks-current-una","errorCode":null,"errorMessage":"{} has no enough internal blocks(current: {}), unable to start recovery. Locations={}","messagePattern":"(.+?) has no enough internal blocks\\(current: (.+?)\\), unable to start recovery\\. Locations=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java","lineNumber":556,"sourceCode":"     * true after we support hflush.\n     */\n    @VisibleForTesting\n    long getSafeLength(Map<Long, BlockRecord> syncBlocks) {\n      final int dataBlkNum = ecPolicy.getNumDataUnits();\n      Preconditions.checkArgument(syncBlocks.size() >= dataBlkNum);\n      long[] blockLengths = new long[syncBlocks.size()];\n      int i = 0;\n      for (BlockRecord r : syncBlocks.values()) {\n        ReplicaRecoveryInfo rInfo = r.getReplicaRecoveryInfo();\n        blockLengths[i++] = rInfo.getNumBytes();\n      }\n      return StripedBlockUtil.getSafeLength(ecPolicy, blockLengths);\n    }\n\n    private void checkLocations(int locationCount)\n        throws IOException {\n      if (locationCount < ecPolicy.getNumDataUnits()) {\n        throw new IOException(block + \" has no enough internal blocks(current: \" + locationCount +\n            \"), unable to start recovery. Locations=\" + Arrays.asList(locs));\n      }\n    }\n  }\n\n  private DatanodeID getDatanodeID(String bpid) throws IOException {\n    BPOfferService bpos = datanode.getBPOfferService(bpid);\n    if (bpos == null) {\n      throw new IOException(\"No block pool offer service for bpid=\" + bpid);\n    }\n    return new DatanodeID(bpos.bpRegistration);\n  }\n\n  private static void logRecoverBlock(String who, RecoveringBlock rb) {\n    ExtendedBlock block = rb.getBlock();\n    DatanodeInfo[] targets = rb.getLocations();\n\n    LOG.info(\"BlockRecoveryWorker: {} calls recoverBlock({}, targets=[{}], newGenerationStamp={}\"","sourceCodeStart":538,"sourceCodeEnd":574,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L538-L574","documentation":"For erasure-coded (striped) blocks, BlockRecoveryWorker.checkLocations requires at least ecPolicy.getNumDataUnits() live internal-block locations before recovery can start: with fewer than the data-unit count, the stripe cannot be reconstructed even using all parity. The IOException aborts recovery for that block immediately, which for EC means the file is at real risk of data loss.","triggerScenarios":"The datanode receives a recoverBlocks command for a striped RecoveringBlock whose locs array has fewer entries than the EC policy's data units (e.g. RS(6,3) with fewer than 6 internal blocks available). Happens after simultaneous loss of more datanodes than the parity count, or when locations were dropped between NN scheduling and DN execution.","commonSituations":"More datanodes (or their disks) lost than the erasure-code parity can cover; decommissioning several datanodes of an EC cluster at once; EC policy reconfigured to a higher data-unit count than surviving replicas can satisfy.","solutions":["Restore the dead/unreachable datanodes or their disks immediately — with EC the block only survives while at least numDataUnits internal blocks remain readable.","Once nodes return, trigger recovery: the NN re-schedules block recovery; verify with 'hdfs fsck /path -files -blocks -locations'.","If the lost internal blocks are unrecoverable, restore the affected files from snapshot, distcp backup, or the original source — the stripe cannot be rebuilt.","Prevent recurrence: cap concurrent datanode decommissions/failures to fewer than the parity units and monitor EC block health via fsck."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before striped recovery, check location count vs EC data units\nbyte[] ecPolicy = rb.getErasureCodingPolicy(); // when available\nint dataUnits = ErasureCodingPolicyManager.getInstance()\n    .getPolicy(rb.getBlock()).getNumDataUnits();\nif (rb.getLocations().length < dataUnits) {\n  LOG.error(\"Stripe under-minimum: {} locations < {} data units; do not attempt recovery\",\n      rb.getLocations().length, dataUnits);\n}","typeGuard":null,"tryCatchPattern":"try {\n  worker.recoverBlocks(who, blocks);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"has no enough internal blocks\")) {\n    // unrecoverable stripe: escalate to backup/restore workflow, do not retry\n    alertDataLossRisk(e);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Never decommission or lose more datanodes concurrently than your EC policy's parity count.","Track EC block health with scheduled 'hdfs fsck' and alert when live internal blocks for any stripe approach numDataUnits.","Keep off-cluster backups/snapshots for erasure-coded data — below the data-unit threshold, Hadoop cannot reconstruct the stripe."],"tags":["hdfs","erasure-coding","datanode","block-recovery","data-loss"],"backgroundTag":"erasure-coding-recovery-impossible","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}