apache/hadoop · warning · RetriableException

Observer Node is too far behind: serverStateId = {} clientSt

Error message

Observer Node is too far behind: serverStateId = {} clientStateId = {}

What it means

GlobalStateIdContext.receiveRequestState throws this RetriableException when the client's stateId exceeds the observer's lastAppliedOrWrittenTxId by more than ESTIMATED_TRANSACTIONS_PER_SECOND x clientWaitTime(seconds) x ESTIMATED_SERVER_TIME_MULTIPLIER. The observer has not tailed enough edits to answer at the client's consistency level, so it explicitly asks the client to retry (typically against another node).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java:159

    long serverStateId = getLastSeenStateId();
    long clientStateId = header.getStateId();
    FSNamesystem.LOG.trace("Client State ID= {} and Server State ID= {}",
        clientStateId, serverStateId);

    if (clientStateId > serverStateId &&
        HAServiceState.ACTIVE.equals(namesystem.getState())) {
      FSNamesystem.LOG.warn("The client stateId: {} is greater than "
          + "the server stateId: {} This is unexpected. "
          + "Resetting client stateId to server stateId",
          clientStateId, serverStateId);
      return serverStateId;
    }
    if (HAServiceState.OBSERVER.equals(namesystem.getState()) &&
        clientStateId - serverStateId >
        ESTIMATED_TRANSACTIONS_PER_SECOND
            * TimeUnit.MILLISECONDS.toSeconds(clientWaitTime)
            * ESTIMATED_SERVER_TIME_MULTIPLIER) {
      throw new RetriableException(
          "Observer Node is too far behind: serverStateId = "
              + serverStateId + " clientStateId = " + clientStateId);
    }
    return clientStateId;
  }

  @Override
  public long getLastSeenStateId() {
    // Should not need to call getCorrectLastAppliedOrWrittenTxId()
    // see HDFS-14822.
    return namesystem.getFSImage().getLastAppliedOrWrittenTxId();
  }

  @Override
  public boolean isCoordinatedCall(String protocolName, String methodName) {
    return protocolName.equals(ClientProtocol.class.getCanonicalName())
        && coordinatedMethods.contains(methodName);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the read — ObserverReadProxyProvider retries and falls back to the Active automatically; transient lag usually resolves in seconds
2. Compare observer vs Active LastAppliedOrWrittenTxId in NameNode JMX to confirm the observer is progressing
3. If the observer never catches up, investigate the EditLogTailer / JournalNode connectivity (logs, network, disk) or restart the observer
4. For workloads needing strict read-your-writes, send those reads to the Active instead of the observer

Example fix

// before: single attempt against observer
FileStatus s = dfs.getFileStatus(path);

// after: tolerate RetriableException while observer catches up
FileStatus s = retryOnRetriable(() -> dfs.getFileStatus(path), 5, 500);

<T> T retryOnRetriable(Callable<T> c, int attempts, long backoffMs) throws Exception {
  for (int i = 0; ; i++) {
    try { return c.call(); }
    catch (RetriableException e) {
      if (i == attempts - 1) throw e;
      Thread.sleep(backoffMs << i);
    }
  }
}

// Pre-check observer lag before routing read-heavy workloads
long activeTx = getJmxTxid(activeJmxUrl);        // LastAppliedOrWrittenTxId (Active)
long observerTx = getJmxTxid(observerJmxUrl);    // LastAppliedOrWrittenTxId (Observer)
if (activeTx - observerTx > LAG_TOLERANCE) {
  routeReadsToActive();  // skip observer until it catches up
}

try {
  return readFromObserver(path);
} catch (RetriableException e) {          // observer behind: transient by contract
  return readFromActive(path);           // or back off and retry the observer
}

When it happens

Trigger: Read-your-writes immediately after a write while the observer has not tailed that edit yet; observer edit tailing lagging (slow JournalNode transfer, GC pauses, EditLogTailer delays); a write burst widening the txid gap beyond the wait-time tolerance.

Common situations: Observers used for read scaling during heavy ingest; observer just restarted and still catching up; JournalNode disk or network bottleneck slowing edit fetch.

Related errors

• Observer Node received request without stateId. This mostly
• {} is in observer state. Cannot be failover target
• Cannot transition from 'observer' to 'active'
• Cannot transition from 'active' to 'observer'
• {} does not support method msync

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/56b84d0addb0cb8b. Report an issue: GitHub.