apache/hadoop · error · IOException

Found {} replica(s) for block {} but none is in RWR or bette

Error message

Found {} replica(s) for block {} but none is in RWR or better state. datanodeids={}

What it means

Error "Found {} replica(s) for block {} but none is in RWR or better state. datanodeids={}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:177

              "Recovery for replica {} on data-node {} is already in progress. " +
                  "Recovery id = {} is aborted.", block, id, rBlock.getNewGenerationStamp(), ripE);
          return;
        } catch (IOException e) {
          ++errorCount;
          InterDatanodeProtocol.LOG.warn("Failed to recover block (block={}, datanode={})",
              block, id, e);
        }
      }

      if (errorCount == locs.length) {
        throw new IOException("All datanodes failed: block=" + block
            + ", datanodeids=" + Arrays.asList(locs));
      }

      // None of the replicas reported by DataNodes has the required original
      // state, report the error.
      if (candidateReplicaCnt > 0 && syncList.isEmpty()) {
        throw new IOException("Found " + candidateReplicaCnt +
            " replica(s) for block " + block + " but none is in " +
            ReplicaState.RWR.name() + " or better state. datanodeids=" +
            Arrays.asList(locs));
      }

      syncBlock(syncList);
    }

    /** Block synchronization. */
    void syncBlock(List<BlockRecord> syncList) throws IOException {
      DatanodeProtocolClientSideTranslatorPB nn =
          getActiveNamenodeForBP(block.getBlockPoolId());

      boolean isTruncateRecovery = rBlock.getNewBlock() != null;
      long blockId = (isTruncateRecovery) ?
          rBlock.getNewBlock().getBlockId() : block.getBlockId();

      LOG.info("BlockRecoveryWorker: block={} (length={}),"

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for in-progress writes to complete or restart stalled writers so replicas reach RWR/FINALIZED state, then retry recovery.
2. Check the state of each replica (via fsck or DataNode logs); if replicas are stuck in RBW/RUR, resolve the underlying writer failures.

When it happens

Trigger: During block recovery, replicas of the block exist but none is in RECOVERY_WAITING_BETTER (RWR) or a better state to serve as the recovery source.

Common situations: Replicas exist for the block but none is in a recoverable RWR-or-better state. Triggered by crashes mid-write; check replica states with fsck before retrying recovery.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2921bbabca2e4aa3. Report an issue: GitHub.