apache/hadoop · critical · IOException

{} has no enough internal blocks(current: {}), unable to sta

Error message

{} has no enough internal blocks(current: {}), unable to start recovery. Locations={}

What it means

For erasure-coded (striped) blocks, BlockRecoveryWorker.checkLocations requires at least ecPolicy.getNumDataUnits() live internal-block locations before recovery can start: with fewer than the data-unit count, the stripe cannot be reconstructed even using all parity. The IOException aborts recovery for that block immediately, which for EC means the file is at real risk of data loss.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:556

     * true after we support hflush.
     */
    @VisibleForTesting
    long getSafeLength(Map<Long, BlockRecord> syncBlocks) {
      final int dataBlkNum = ecPolicy.getNumDataUnits();
      Preconditions.checkArgument(syncBlocks.size() >= dataBlkNum);
      long[] blockLengths = new long[syncBlocks.size()];
      int i = 0;
      for (BlockRecord r : syncBlocks.values()) {
        ReplicaRecoveryInfo rInfo = r.getReplicaRecoveryInfo();
        blockLengths[i++] = rInfo.getNumBytes();
      }
      return StripedBlockUtil.getSafeLength(ecPolicy, blockLengths);
    }

    private void checkLocations(int locationCount)
        throws IOException {
      if (locationCount < ecPolicy.getNumDataUnits()) {
        throw new IOException(block + " has no enough internal blocks(current: " + locationCount +
            "), unable to start recovery. Locations=" + Arrays.asList(locs));
      }
    }
  }

  private DatanodeID getDatanodeID(String bpid) throws IOException {
    BPOfferService bpos = datanode.getBPOfferService(bpid);
    if (bpos == null) {
      throw new IOException("No block pool offer service for bpid=" + bpid);
    }
    return new DatanodeID(bpos.bpRegistration);
  }

  private static void logRecoverBlock(String who, RecoveringBlock rb) {
    ExtendedBlock block = rb.getBlock();
    DatanodeInfo[] targets = rb.getLocations();

    LOG.info("BlockRecoveryWorker: {} calls recoverBlock({}, targets=[{}], newGenerationStamp={}"

View on GitHub (pinned to 2add963021)

Solutions

Restore the dead/unreachable datanodes or their disks immediately — with EC the block only survives while at least numDataUnits internal blocks remain readable.
Once nodes return, trigger recovery: the NN re-schedules block recovery; verify with 'hdfs fsck /path -files -blocks -locations'.
If the lost internal blocks are unrecoverable, restore the affected files from snapshot, distcp backup, or the original source — the stripe cannot be rebuilt.
Prevent recurrence: cap concurrent datanode decommissions/failures to fewer than the parity units and monitor EC block health via fsck.

Defensive patterns

Strategy: validation

Validate before calling

// Before striped recovery, check location count vs EC data units
byte[] ecPolicy = rb.getErasureCodingPolicy(); // when available
int dataUnits = ErasureCodingPolicyManager.getInstance()
    .getPolicy(rb.getBlock()).getNumDataUnits();
if (rb.getLocations().length < dataUnits) {
  LOG.error("Stripe under-minimum: {} locations < {} data units; do not attempt recovery",
      rb.getLocations().length, dataUnits);
}

Try / catch

try {
  worker.recoverBlocks(who, blocks);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("has no enough internal blocks")) {
    // unrecoverable stripe: escalate to backup/restore workflow, do not retry
    alertDataLossRisk(e);
  } else {
    throw e;
  }
}

Prevention

Never decommission or lose more datanodes concurrently than your EC policy's parity count.
Track EC block health with scheduled 'hdfs fsck' and alert when live internal blocks for any stripe approach numDataUnits.
Keep off-cluster backups/snapshots for erasure-coded data — below the data-unit threshold, Hadoop cannot reconstruct the stripe.

When it happens

Trigger: The datanode receives a recoverBlocks command for a striped RecoveringBlock whose locs array has fewer entries than the EC policy's data units (e.g. RS(6,3) with fewer than 6 internal blocks available). Happens after simultaneous loss of more datanodes than the parity count, or when locations were dropped between NN scheduling and DN execution.

Common situations: More datanodes (or their disks) lost than the erasure-code parity can cover; decommissioning several datanodes of an EC cluster at once; EC policy reconfigured to a higher data-unit count than surviving replicas can satisfy.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/999633b2875f4c33. Report an issue: GitHub.