{"record":{"id":"56b84d0addb0cb8b","repo":"apache/hadoop","slug":"observer-node-is-too-far-behind-serverstateid","errorCode":null,"errorMessage":"Observer Node is too far behind: serverStateId = {} clientStateId = {}","messagePattern":"Observer Node is too far behind: serverStateId = (.+?) clientStateId = (.+?)","errorType":"exception","errorClass":"RetriableException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java","lineNumber":159,"sourceCode":"    long serverStateId = getLastSeenStateId();\n    long clientStateId = header.getStateId();\n    FSNamesystem.LOG.trace(\"Client State ID= {} and Server State ID= {}\",\n        clientStateId, serverStateId);\n\n    if (clientStateId > serverStateId &&\n        HAServiceState.ACTIVE.equals(namesystem.getState())) {\n      FSNamesystem.LOG.warn(\"The client stateId: {} is greater than \"\n          + \"the server stateId: {} This is unexpected. \"\n          + \"Resetting client stateId to server stateId\",\n          clientStateId, serverStateId);\n      return serverStateId;\n    }\n    if (HAServiceState.OBSERVER.equals(namesystem.getState()) &&\n        clientStateId - serverStateId >\n        ESTIMATED_TRANSACTIONS_PER_SECOND\n            * TimeUnit.MILLISECONDS.toSeconds(clientWaitTime)\n            * ESTIMATED_SERVER_TIME_MULTIPLIER) {\n      throw new RetriableException(\n          \"Observer Node is too far behind: serverStateId = \"\n              + serverStateId + \" clientStateId = \" + clientStateId);\n    }\n    return clientStateId;\n  }\n\n  @Override\n  public long getLastSeenStateId() {\n    // Should not need to call getCorrectLastAppliedOrWrittenTxId()\n    // see HDFS-14822.\n    return namesystem.getFSImage().getLastAppliedOrWrittenTxId();\n  }\n\n  @Override\n  public boolean isCoordinatedCall(String protocolName, String methodName) {\n    return protocolName.equals(ClientProtocol.class.getCanonicalName())\n        && coordinatedMethods.contains(methodName);\n  }","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java#L141-L177","documentation":"GlobalStateIdContext.receiveRequestState throws this RetriableException when the client's stateId exceeds the observer's lastAppliedOrWrittenTxId by more than ESTIMATED_TRANSACTIONS_PER_SECOND x clientWaitTime(seconds) x ESTIMATED_SERVER_TIME_MULTIPLIER. The observer has not tailed enough edits to answer at the client's consistency level, so it explicitly asks the client to retry (typically against another node).","triggerScenarios":"Read-your-writes immediately after a write while the observer has not tailed that edit yet; observer edit tailing lagging (slow JournalNode transfer, GC pauses, EditLogTailer delays); a write burst widening the txid gap beyond the wait-time tolerance.","commonSituations":"Observers used for read scaling during heavy ingest; observer just restarted and still catching up; JournalNode disk or network bottleneck slowing edit fetch.","solutions":["Retry the read — ObserverReadProxyProvider retries and falls back to the Active automatically; transient lag usually resolves in seconds","Compare observer vs Active LastAppliedOrWrittenTxId in NameNode JMX to confirm the observer is progressing","If the observer never catches up, investigate the EditLogTailer / JournalNode connectivity (logs, network, disk) or restart the observer","For workloads needing strict read-your-writes, send those reads to the Active instead of the observer"],"exampleFix":"// before: single attempt against observer\nFileStatus s = dfs.getFileStatus(path);\n\n// after: tolerate RetriableException while observer catches up\nFileStatus s = retryOnRetriable(() -> dfs.getFileStatus(path), 5, 500);\n\n<T> T retryOnRetriable(Callable<T> c, int attempts, long backoffMs) throws Exception {\n  for (int i = 0; ; i++) {\n    try { return c.call(); }\n    catch (RetriableException e) {\n      if (i == attempts - 1) throw e;\n      Thread.sleep(backoffMs << i);\n    }\n  }\n}","handlingStrategy":"retry","validationCode":"// Pre-check observer lag before routing read-heavy workloads\nlong activeTx = getJmxTxid(activeJmxUrl);        // LastAppliedOrWrittenTxId (Active)\nlong observerTx = getJmxTxid(observerJmxUrl);    // LastAppliedOrWrittenTxId (Observer)\nif (activeTx - observerTx > LAG_TOLERANCE) {\n  routeReadsToActive();  // skip observer until it catches up\n}","typeGuard":null,"tryCatchPattern":"try {\n  return readFromObserver(path);\n} catch (RetriableException e) {          // observer behind: transient by contract\n  return readFromActive(path);           // or back off and retry the observer\n}","preventionTips":["Monitor observer-vs-Active txid lag (LastAppliedOrWrittenTxid in JMX) and alert on sustained gaps","Keep EditLogTailer healthy: watch for tailer errors and JournalNode throughput","Use ObserverReadProxyProvider so RetriableException is retried/rerouted without application code"],"tags":["hdfs","observer","ha","replication-lag","retry","read-your-writes"],"backgroundTag":"read-replica-lag","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}