{"record":{"id":"4a0566857402f74f","repo":"apache/hadoop","slug":"observer-node-received-request-without-stateid-th","errorCode":null,"errorMessage":"Observer Node received request without stateId. This mostly likely is because client is not configured with ObserverReadProxyProvider","messagePattern":"Observer Node received request without stateId\\. This mostly likely is because client is not configured with ObserverReadProxyProvider","errorType":"exception","errorClass":"StandbyException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java","lineNumber":137,"sourceCode":"   *\n   * @param header The RPC request header.\n   * @param clientWaitTime time in milliseconds indicating how long client\n   *    waits for the server response. It is used to verify if the client's\n   *    state is too far ahead of the server's\n   * @return the minimum of the state ids of the client or the server.\n   * @throws RetriableException if Observer is too far behind.\n   */\n  @Override\n  public long receiveRequestState(RpcRequestHeaderProto header,\n      long clientWaitTime) throws IOException {\n    if (!header.hasStateId() &&\n        HAServiceState.OBSERVER.equals(namesystem.getState())) {\n      // This could happen if client configured with non-observer proxy provider\n      // (e.g., ConfiguredFailoverProxyProvider) is accessing a cluster with\n      // observers. In this case, we should let the client failover to the\n      // active node, rather than potentially serving stale result (client\n      // stateId is 0 if not set).\n      throw new StandbyException(\"Observer Node received request without \"\n          + \"stateId. This mostly likely is because client is not configured \"\n          + \"with \" + ObserverReadProxyProvider.class.getSimpleName());\n    }\n    long serverStateId = getLastSeenStateId();\n    long clientStateId = header.getStateId();\n    FSNamesystem.LOG.trace(\"Client State ID= {} and Server State ID= {}\",\n        clientStateId, serverStateId);\n\n    if (clientStateId > serverStateId &&\n        HAServiceState.ACTIVE.equals(namesystem.getState())) {\n      FSNamesystem.LOG.warn(\"The client stateId: {} is greater than \"\n          + \"the server stateId: {} This is unexpected. \"\n          + \"Resetting client stateId to server stateId\",\n          clientStateId, serverStateId);\n      return serverStateId;\n    }\n    if (HAServiceState.OBSERVER.equals(namesystem.getState()) &&\n        clientStateId - serverStateId >","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java#L119-L155","documentation":"An Observer NameNode throws this StandbyException from GlobalStateIdContext.receiveRequestState when a client request carries no stateId (header.hasStateId() is false). Without a client stateId the observer cannot guarantee read-your-writes, so it rejects the request to make the client's failover proxy reroute to the Active node. It nearly always means the client is configured with a non-observer provider such as ConfiguredFailoverProxyProvider against a namespace that has Observer nodes.","triggerScenarios":"Client hdfs-site.xml sets dfs.client.failover.proxy.provider.<nameservice>=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider (or RouterOfActive) and a read lands on an Observer; a custom proxy provider that does not attach stateId to the RPC header; pre-3.2 clients that predate client stateId (HDFS-14581) hitting an observer.","commonSituations":"Observers enabled in an HA cluster while existing clients keep their old failover provider config; mixed client versions during a rolling upgrade to Hadoop 3.2+; homegrown RPC wrappers that strip protocol headers.","solutions":["Set dfs.client.failover.proxy.provider.<nameservice>=org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider in client hdfs-site.xml and restart jobs","If those clients must not use observers, serve them only Active/Standby addresses (e.g. via a Router or separate nameservice) so requests never reach the observer","For custom proxy providers, forward the stateId received in responses into subsequent request headers"],"exampleFix":"<!-- before: client hdfs-site.xml -->\n<property>\n  <name>dfs.client.failover.proxy.provider.myns</name>\n  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>\n</property>\n\n<!-- after -->\n<property>\n  <name>dfs.client.failover.proxy.provider.myns</name>\n  <value>org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider</value>\n</property>","handlingStrategy":"validation","validationCode":"// Client startup check: fail fast on a provider that cannot talk to observers\nConfiguration conf = new Configuration();\nString ns = conf.get(DFSConfigKeys.DFS_NAMESERVICE_ID);\nClass<?> provider = conf.getClass(\n    DFSConfigKeys.DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + \".\" + ns, null);\nif (provider != null\n    && !ObserverReadProxyProvider.class.isAssignableFrom(provider)) {\n  LOG.warn(\"Clients using {} will be rejected by Observer nodes; \"\n      + \"configure ObserverReadProxyProvider to use observers\",\n      provider.getName());\n}","typeGuard":null,"tryCatchPattern":"catch (StandbyException e) {\n  // Expected from Observers for stateId-less clients; the failover layer\n  // should reroute to the Active. If it bubbles up, the provider config is wrong.\n  LOG.debug(\"Rerouted away from observer: {}\", e.getMessage());\n}","preventionTips":["Standardize dfs.client.failover.proxy.provider.<ns> on ObserverReadProxyProvider when observers exist","Keep clients at Hadoop 3.2+ so stateId is always sent","Alert on StandbyException clusters in client logs — they indicate provider/config drift, not transient faults"],"tags":["hdfs","ha","observer","client-configuration","proxy-provider","read-your-writes"],"backgroundTag":"failover-proxy-provider-misconfiguration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}