apache/hadoop · warning · BPServiceActorActionException

Failed to report bad block {} to namenode.

Error message

Failed to report bad block {} to namenode.

What it means

ReportBadBlockAction.execute() calls bpNamenode.reportBadBlocks(locatedBlock) to tell the NameNode a block replica is corrupt. A RemoteException (an error raised BY the NameNode, e.g. 'replica is not corrupt' or 'block does not exist') is only logged at info level, but any other IOException (transport failure, timeout, NN unreachable) is wrapped in BPServiceActorActionException so the BPOfferService action queue can retry it later.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java:70

  public void reportTo(DatanodeProtocolClientSideTranslatorPB bpNamenode, 
    DatanodeRegistration bpRegistration) throws BPServiceActorActionException {
    if (bpRegistration == null) {
      return;
    }
    DatanodeInfo[] dnArr = {new DatanodeInfoBuilder()
        .setNodeID(bpRegistration).build()};
    String[] uuids = { storageUuid };
    StorageType[] types = { storageType };
    LocatedBlock[] locatedBlock = { new LocatedBlock(block,
        dnArr, uuids, types) };

    try {
      bpNamenode.reportBadBlocks(locatedBlock);
    } catch (RemoteException re) {
      DataNode.LOG.info("reportBadBlock encountered RemoteException for "
          + "block:  " + block , re);
    } catch (IOException e) {
      throw new BPServiceActorActionException("Failed to report bad block "
          + block + " to namenode.", e);
    }
  }

  @Override
  public int hashCode() {
    final int prime = 31;
    int result = 1;
    result = prime * result + ((block == null) ? 0 : block.hashCode());
    result = prime * result
        + ((storageType == null) ? 0 : storageType.hashCode());
    result = prime * result
        + ((storageUuid == null) ? 0 : storageUuid.hashCode());
    return result;
  }

  @Override
  public boolean equals(Object obj) {

View on GitHub (pinned to 2add963021)

Solutions

1. Verify NameNode reachability and health (its UI/RPC port) - the action is already retried by the BPServiceActor thread, so a transient NN outage self-heals
2. Check for active-standby flip-flop or split-brain causing repeated RPC failures
3. If it persists, capture the cause in BPServiceActorActionException and inspect the underlying IOException (timeout vs connection refused) to target the real network/config issue

import org.apache.hadoop.ha.HAServiceProtocol;
// before executing bad-block reports, confirm the NN is up and active
boolean nnReady = dfsCluster != null && haServiceState == HAServiceProtocol.HAServiceState.ACTIVE;

catch (BPServiceActorActionException e) {
  // action stays queued and retried by BPOfferService; log at info, do not crash the actor thread
  LOG.info("Bad block report deferred, will retry: {}", e.getMessage());
}

When it happens

Trigger: The DataNode's DataNodeFaultInjector or a checksum failure queues a ReportBadBlockAction; when the queued action runs, the RPC to the NameNode fails with a non-RemoteException IOException - NameNode down/restarting, network partition, RPC queue full/timeout, HA failover in progress.

Common situations: Transient NameNode unavailability during HA failover while the DN reports corrupt blocks; network issues between DN and NN; the same bad block reported repeatedly because earlier reports never landed.

Related errors

• Unsupported protocol found when creating the proxy connectio
• Replica was found but missing fields.
• Datanode {datanode} not found.
• Datanode denied communication with namenode because the host
• Cannot lock storage {root}. The directory is already locked

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f54604e452b3a66. Report an issue: GitHub.