apache/hadoop · error · IllegalArgumentException

Unexpected HAServiceStateProto:

Error message

Unexpected HAServiceStateProto:

What it means

Converts the state field of the NameNode HA heartbeat (NNHAStatusHeartbeatProto.State, sent from the active NN to standby/backup nodes) into HAServiceState. Only ACTIVE, STANDBY and OBSERVER are understood; any other enum value on the wire fails with IllegalArgumentException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java:788

    if(state != null) {
      builder.setState(convert(info.getState()));
    }
    return builder.build();
  }

  public static HAServiceState convert(NNHAStatusHeartbeatProto.State s) {
    if (s == null) {
      return null;
    }
    switch (s) {
    case ACTIVE:
      return HAServiceState.ACTIVE;
    case STANDBY:
      return HAServiceState.STANDBY;
    case OBSERVER:
      return HAServiceState.OBSERVER;
    default:
      throw new IllegalArgumentException("Unexpected HAServiceStateProto:"
          + s);
    }
  }

  public static NNHAStatusHeartbeatProto.State convert(HAServiceState s) {
    if (s == null) {
      return null;
    }
    switch (s) {
    case ACTIVE:
      return NNHAStatusHeartbeatProto.State.ACTIVE;
    case STANDBY:
      return NNHAStatusHeartbeatProto.State.STANDBY;
    case OBSERVER:
      return NNHAStatusHeartbeatProto.State.OBSERVER;
    default:
      throw new IllegalArgumentException("Unexpected HAServiceState:"
          + s);

View on GitHub (pinned to 2add963021)

Solutions

1. Upgrade the node that parses the heartbeat (standby/backup NN) to at least the sender's release.
2. Note the unknown value from the exception message to identify the sender's version.
3. Finish or revert the rolling upgrade so both HA peers run the same version.

import org.apache.hadoop.ha.HAServiceState;
import java.util.EnumSet;

static boolean isKnownHeartbeatState(NNHAStatusHeartbeatProto.State s) {
  return EnumSet.of(
      NNHAStatusHeartbeatProto.State.ACTIVE,
      NNHAStatusHeartbeatProto.State.STANDBY,
      NNHAStatusHeartbeatProto.State.OBSERVER).contains(s);
}
// guard before converting
if (!isKnownHeartbeatState(proto.getState())) {
  LOG.warn("Unknown HA heartbeat state from peer: {}", proto.getState());
  return;
}

static HAServiceState toHAServiceStateOrNull(NNHAStatusHeartbeatProto.State s) {
  switch (s) {
    case ACTIVE: return HAServiceState.ACTIVE;
    case STANDBY: return HAServiceState.STANDBY;
    case OBSERVER: return HAServiceState.OBSERVER;
    default: return null; // caller skips instead of crashing
  }
}

try {
  HAServiceState st = PBHelper.convert(proto.getState());
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("HAServiceStateProto")) {
    // peer is newer than us: log and ignore the heartbeat until upgraded
    LOG.warn("Ignoring heartbeat with unknown state: {}", e.getMessage());
    return;
  }
  throw e;
}

When it happens

Trigger: A heartbeat arrives carrying a state constant this build does not know — e.g. an older standby/backup NameNode receiving a state introduced by a newer active (OBSERVER support or later additions), typically mid rolling upgrade.

Common situations: Mixed-version HA clusters during rolling upgrades; a standby older than the active; replaying recorded RPC traffic from a newer release in tests.

Related errors

• Unexpected HAServiceState:
• Unrecognized section {}
• Unsupported file version {}
• {} does not support method msync
• Not able to initialize fs in getDefaultBlockSize for path <

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dc41b8d2832d8905. Report an issue: GitHub.