apache/hadoop · critical · IOException

{} missing blocks, the stripe is: {}; locatedBlocks is: {}

Error message

{} missing blocks, the stripe is: {}; locatedBlocks is: {}

What it means

During a striped (erasure-coded) read, StripeReader compares alignedStripe.missingChunksNum to parityBlkNum; erasure coding can reconstruct at most parity-count missing chunks, so more missing chunks than parity makes the stripe undecodable and an IOException listing the stripe and locatedBlocks is thrown. This is read-path data unavailability: either DataNodes are (transiently) down beyond the policy's tolerance or the stripe has genuinely lost data.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java:178

  abstract void decode() throws IOException;

  /*
   * Default close do nothing.
   */
  void close() {
  }

  void updateState4SuccessRead(StripingChunkReadResult result) {
    Preconditions.checkArgument(
        result.state == StripingChunkReadResult.SUCCESSFUL);
    readerInfos[result.index].setOffset(alignedStripe.getOffsetInBlock()
        + alignedStripe.getSpanInBlock());
  }

  private void checkMissingBlocks() throws IOException {
    if (alignedStripe.missingChunksNum > parityBlkNum) {
      clearFutures();
      throw new IOException(alignedStripe.missingChunksNum
          + " missing blocks, the stripe is: " + alignedStripe
          + "; locatedBlocks is: " + dfsStripedInputStream.getLocatedBlocks());
    }
  }

  /**
   * We need decoding. Thus go through all the data chunks and make sure we
   * submit read requests for all of them.
   */
  private void readDataForDecoding() throws IOException {
    prepareDecodeInputs();
    for (int i = 0; i < dataBlkNum; i++) {
      Preconditions.checkNotNull(alignedStripe.chunks[i]);
      if (alignedStripe.chunks[i].state == StripingChunk.REQUESTED) {
        if (!readChunk(targetBlocks[i], i)) {
          alignedStripe.missingChunksNum++;
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Check DataNode liveness (hdfs dfsadmin -report) and restart/restore failed nodes — within-parity loss recovers automatically once nodes return
2. Run hdfs fsck /path -files -blocks -locations -replicaDetails to see whether the stripe is transiently degraded or permanently damaged
3. If fsck reports actual missing/corrupt chunks beyond parity, restore the file from its source (distcp/re-ingest) — the data cannot be reconstructed client-side
4. Prevent recurrence: spread EC block group placement across racks and avoid batching DataNode restarts within one placement set

// Pre-flight: ensure available DataNodes exceed the EC policy's parity count
// before striped reads: hdfs dfsadmin -report -> Live nodes vs policy width
// hdfs fsck /path -files -blocks -locations should show no missing chunks

try {
  in.read(buffer, off, len);
} catch (IOException e) {
  if (e.getMessage().contains("missing blocks, the stripe is")) {
    // transient beyond-parity unavailability: wait for DN recovery, re-open and re-read
    waitForDatanodes(conf, timeout);
    reopenAndSeek(in.getPosition());
  } else throw e;
}

When it happens

Trigger: Positional/sequential read of an EC file (e.g., RS-6-3, RS-10-4) while more DataNodes/chunks in one stripe are unavailable than the parity count: multiple simultaneous DataNode failures, corrupted chunks, or decommissioned nodes not yet re-replicated; also seen on under-constructed EC block groups.

Common situations: EC datasets during rolling maintenance when several DataNodes in one stripe's placement set restart together; real hardware loss exceeding parity; heavy-load windows where chunk reads fail and push missing count over parity; files written with an EC policy the cluster lacks disks to satisfy.

Related errors

• {} has no enough internal blocks(current: {}), unable to sta
• Unknown BlockChecksumType: " + groupChecksumType
• {blockURI}
• Block data not found, r={r}
• Fail to get block checksum for {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/743f0512fa48a787. Report an issue: GitHub.