{"record":{"id":"743f0512fa48a787","repo":"apache/hadoop","slug":"missing-blocks-the-stripe-is-locatedblock","errorCode":null,"errorMessage":"{} missing blocks, the stripe is: {}; locatedBlocks is: {}","messagePattern":"(.+?) missing blocks, the stripe is: (.+?); locatedBlocks is: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java","lineNumber":178,"sourceCode":"  abstract void decode() throws IOException;\n\n  /*\n   * Default close do nothing.\n   */\n  void close() {\n  }\n\n  void updateState4SuccessRead(StripingChunkReadResult result) {\n    Preconditions.checkArgument(\n        result.state == StripingChunkReadResult.SUCCESSFUL);\n    readerInfos[result.index].setOffset(alignedStripe.getOffsetInBlock()\n        + alignedStripe.getSpanInBlock());\n  }\n\n  private void checkMissingBlocks() throws IOException {\n    if (alignedStripe.missingChunksNum > parityBlkNum) {\n      clearFutures();\n      throw new IOException(alignedStripe.missingChunksNum\n          + \" missing blocks, the stripe is: \" + alignedStripe\n          + \"; locatedBlocks is: \" + dfsStripedInputStream.getLocatedBlocks());\n    }\n  }\n\n  /**\n   * We need decoding. Thus go through all the data chunks and make sure we\n   * submit read requests for all of them.\n   */\n  private void readDataForDecoding() throws IOException {\n    prepareDecodeInputs();\n    for (int i = 0; i < dataBlkNum; i++) {\n      Preconditions.checkNotNull(alignedStripe.chunks[i]);\n      if (alignedStripe.chunks[i].state == StripingChunk.REQUESTED) {\n        if (!readChunk(targetBlocks[i], i)) {\n          alignedStripe.missingChunksNum++;\n        }\n      }","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java#L160-L196","documentation":"During a striped (erasure-coded) read, StripeReader compares alignedStripe.missingChunksNum to parityBlkNum; erasure coding can reconstruct at most parity-count missing chunks, so more missing chunks than parity makes the stripe undecodable and an IOException listing the stripe and locatedBlocks is thrown. This is read-path data unavailability: either DataNodes are (transiently) down beyond the policy's tolerance or the stripe has genuinely lost data.","triggerScenarios":"Positional/sequential read of an EC file (e.g., RS-6-3, RS-10-4) while more DataNodes/chunks in one stripe are unavailable than the parity count: multiple simultaneous DataNode failures, corrupted chunks, or decommissioned nodes not yet re-replicated; also seen on under-constructed EC block groups.","commonSituations":"EC datasets during rolling maintenance when several DataNodes in one stripe's placement set restart together; real hardware loss exceeding parity; heavy-load windows where chunk reads fail and push missing count over parity; files written with an EC policy the cluster lacks disks to satisfy.","solutions":["Check DataNode liveness (hdfs dfsadmin -report) and restart/restore failed nodes — within-parity loss recovers automatically once nodes return","Run hdfs fsck /path -files -blocks -locations -replicaDetails to see whether the stripe is transiently degraded or permanently damaged","If fsck reports actual missing/corrupt chunks beyond parity, restore the file from its source (distcp/re-ingest) — the data cannot be reconstructed client-side","Prevent recurrence: spread EC block group placement across racks and avoid batching DataNode restarts within one placement set"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: ensure available DataNodes exceed the EC policy's parity count\n// before striped reads: hdfs dfsadmin -report -> Live nodes vs policy width\n// hdfs fsck /path -files -blocks -locations should show no missing chunks","typeGuard":null,"tryCatchPattern":"try {\n  in.read(buffer, off, len);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"missing blocks, the stripe is\")) {\n    // transient beyond-parity unavailability: wait for DN recovery, re-open and re-read\n    waitForDatanodes(conf, timeout);\n    reopenAndSeek(in.getPosition());\n  } else throw e;\n}","preventionTips":["Keep simultaneous EC DataNode failures under the parity count; stagger restarts across stripe placement sets","Monitor with fsck and NameNode missing-blocks metrics; alert before parity is exhausted","Choose EC policy width/parity matching your maintenance patterns (RS-10-4 tolerates 4 failures)"],"tags":["hdfs","erasure-coding","data-loss","datanode","striped-read"],"backgroundTag":"erasure-coded-data-loss","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}