{"record":{"id":"23a7614b1d6b2d17","repo":"apache/hadoop","slug":"processing-rpc-request-caught","errorCode":null,"errorMessage":"Processing RPC request caught ","messagePattern":"Processing RPC request caught ","errorType":"exception","errorClass":"RpcServerException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java","lineNumber":3051,"sourceCode":"          String methodName;\n          String protoName;\n          ProtobufRpcEngine2.RpcProtobufRequest req =\n              (ProtobufRpcEngine2.RpcProtobufRequest) call.rpcRequest;\n          try {\n            methodName = req.getRequestHeader().getMethodName();\n            protoName = req.getRequestHeader().getDeclaringClassProtocolName();\n            if (alignmentContext.isCoordinatedCall(protoName, methodName)) {\n              call.markCallCoordinated(true);\n              long stateId;\n              stateId = alignmentContext.receiveRequestState(\n                  header, getMaxIdleTime());\n              call.setClientStateId(stateId);\n              if (header.hasRouterFederatedState()) {\n                call.setFederatedNamespaceState(header.getRouterFederatedState());\n              }\n            }\n          } catch (IOException ioe) {\n            throw new RpcServerException(\"Processing RPC request caught \", ioe);\n          }\n        }\n\n        try {\n          internalQueueCall(call);\n        } catch (RpcServerException rse) {\n          throw rse;\n        } catch (IOException ioe) {\n          throw new FatalRpcServerException(\n              RpcErrorCodeProto.ERROR_RPC_SERVER, ioe);\n        }\n        incRpcCount();  // Increment the rpc count\n      } finally {\n        AuthorizationContext.clear();\n      }\n    }\n\n    /**","sourceCodeStart":3033,"sourceCodeEnd":3069,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L3033-L3069","documentation":"While preparing a coordinated call, the server validates the client's stateId through alignmentContext.receiveRequestState(); any IOException it throws is wrapped as RpcServerException('Processing RPC request caught '). With Observer reads / Router federated state (GlobalStateIdContext), this fires when an Observer NameNode gets a request without a stateId (client not using ObserverReadProxyProvider) or when the Observer is too far behind the client (RetriableException).","triggerScenarios":"A client uses ConfiguredFailoverProxyProvider against a cluster with Observer NameNodes, so requests carry no stateId; an Observer's stateId lags the client's beyond the estimated catch-up window (clientStateId - serverStateId > threshold); Router federated state alignment detects inconsistent state across namespaces.","commonSituations":"Enabling dfs.internal.nameservices / Observer reads (RBF or Observer NameNode) while clients keep old proxy providers; observers falling behind after a large edit-log burst or checkpoint lag; failover races where a client's cached stateId is ahead of a freshly promoted node.","solutions":["Configure clients to use ObserverReadProxyProvider (dfs.client.failover.proxy.provider.<ns>=...ObserverReadProxyProvider) when Observer reads are enabled","RetriableException is retriable: let the client retry (retry policies / multi-proxy failover to another Observer or the Active) instead of failing the job","If Observers lag persistently, investigate edit-log tailing/checkpointing on the Observer so its stateId catches up","For Router federation, verify state store sync and router federated state configuration (see HDFS Router state alignment docs)"],"exampleFix":"// before: client without observer support\nconf.set(\"dfs.client.failover.proxy.provider.\" + ns,\n    \"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider\");\n\n// after: observer-aware provider so requests carry a stateId\nconf.set(\"dfs.client.failover.proxy.provider.\" + ns,\n    \"org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider\");","handlingStrategy":"retry","validationCode":"// client: enable observer-aware failover before opening connections\nconf.set(\"dfs.client.failover.proxy.provider.\" + ns,\n    \"org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider\");","typeGuard":null,"tryCatchPattern":"try {\n  return proxy.getMtimeInfo(path);\n} catch (RemoteException re) {\n  if (re.getClassName().contains(\"RetriableException\")\n      || re.getClassName().contains(\"StandbyException\")) {\n    // Observer behind or stateId missing: retry via failover policy / Active NN\n    return retryWithFailover(() -> proxy.getMtimeInfo(path));\n  } else { throw re; }\n}","preventionTips":["Use ObserverReadProxyProvider whenever Observer reads are enabled","Let RetriableException drive client retries; do not convert it to a fatal failure","Monitor Observer stateId lag and edit-tail health to keep observers within the catch-up window"],"tags":["rpc","observer-namenode","state-alignment","hdfs","router"],"backgroundTag":"state-id-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}