apache/hadoop · warning · StandbyException

Observer Node received request without stateId. This mostly

Error message

Observer Node received request without stateId. This mostly likely is because client is not configured with ObserverReadProxyProvider

What it means

An Observer NameNode throws this StandbyException from GlobalStateIdContext.receiveRequestState when a client request carries no stateId (header.hasStateId() is false). Without a client stateId the observer cannot guarantee read-your-writes, so it rejects the request to make the client's failover proxy reroute to the Active node. It nearly always means the client is configured with a non-observer provider such as ConfiguredFailoverProxyProvider against a namespace that has Observer nodes.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java:137

   *
   * @param header The RPC request header.
   * @param clientWaitTime time in milliseconds indicating how long client
   *    waits for the server response. It is used to verify if the client's
   *    state is too far ahead of the server's
   * @return the minimum of the state ids of the client or the server.
   * @throws RetriableException if Observer is too far behind.
   */
  @Override
  public long receiveRequestState(RpcRequestHeaderProto header,
      long clientWaitTime) throws IOException {
    if (!header.hasStateId() &&
        HAServiceState.OBSERVER.equals(namesystem.getState())) {
      // This could happen if client configured with non-observer proxy provider
      // (e.g., ConfiguredFailoverProxyProvider) is accessing a cluster with
      // observers. In this case, we should let the client failover to the
      // active node, rather than potentially serving stale result (client
      // stateId is 0 if not set).
      throw new StandbyException("Observer Node received request without "
          + "stateId. This mostly likely is because client is not configured "
          + "with " + ObserverReadProxyProvider.class.getSimpleName());
    }
    long serverStateId = getLastSeenStateId();
    long clientStateId = header.getStateId();
    FSNamesystem.LOG.trace("Client State ID= {} and Server State ID= {}",
        clientStateId, serverStateId);

    if (clientStateId > serverStateId &&
        HAServiceState.ACTIVE.equals(namesystem.getState())) {
      FSNamesystem.LOG.warn("The client stateId: {} is greater than "
          + "the server stateId: {} This is unexpected. "
          + "Resetting client stateId to server stateId",
          clientStateId, serverStateId);
      return serverStateId;
    }
    if (HAServiceState.OBSERVER.equals(namesystem.getState()) &&
        clientStateId - serverStateId >

View on GitHub (pinned to 2add963021)

Solutions

1. Set dfs.client.failover.proxy.provider.<nameservice>=org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider in client hdfs-site.xml and restart jobs
2. If those clients must not use observers, serve them only Active/Standby addresses (e.g. via a Router or separate nameservice) so requests never reach the observer
3. For custom proxy providers, forward the stateId received in responses into subsequent request headers

Example fix

<!-- before: client hdfs-site.xml -->
<property>
  <name>dfs.client.failover.proxy.provider.myns</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<!-- after -->
<property>
  <name>dfs.client.failover.proxy.provider.myns</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider</value>
</property>

// Client startup check: fail fast on a provider that cannot talk to observers
Configuration conf = new Configuration();
String ns = conf.get(DFSConfigKeys.DFS_NAMESERVICE_ID);
Class<?> provider = conf.getClass(
    DFSConfigKeys.DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + "." + ns, null);
if (provider != null
    && !ObserverReadProxyProvider.class.isAssignableFrom(provider)) {
  LOG.warn("Clients using {} will be rejected by Observer nodes; "
      + "configure ObserverReadProxyProvider to use observers",
      provider.getName());
}

catch (StandbyException e) {
  // Expected from Observers for stateId-less clients; the failover layer
  // should reroute to the Active. If it bubbles up, the provider config is wrong.
  LOG.debug("Rerouted away from observer: {}", e.getMessage());
}

When it happens

Trigger: Client hdfs-site.xml sets dfs.client.failover.proxy.provider.<nameservice>=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider (or RouterOfActive) and a read lands on an Observer; a custom proxy provider that does not attach stateId to the RPC header; pre-3.2 clients that predate client stateId (HDFS-14581) hitting an observer.

Common situations: Observers enabled in an HA cluster while existing clients keep their old failover provider config; mixed client versions during a rolling upgrade to Hadoop 3.2+; homegrown RPC wrappers that strip protocol headers.

Related errors

• Observer Node is too far behind: serverStateId = {} clientSt
• {} is in observer state. Cannot be failover target
• Cannot transition from 'observer' to 'active'
• Cannot transition from 'active' to 'observer'
• {} does not support method msync

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4a0566857402f74f. Report an issue: GitHub.