apache/hadoop · warning · RetriableException

Router is in startup mode

Error message

Router is in startup mode

What it means

RouterWebHdfsMethods.getRpcClientProtocol throws RetriableException when a WebHDFS request reaches the Router before its RPC server has been created (router.getRpcServer() returns null). It signals a startup race, not a permanent failure: the HTTP endpoint (Router WebHDFS) is up while the core RPC service is still initializing. RetriableException tells well-behaved clients to retry the request later.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java:155

  }

  @Override
  protected void init(final UserGroupInformation ugi,
                      final DelegationParam delegation,
                      final UserParam username, final DoAsParam doAsUser,
                      final UriFsPathParam path, final HttpOpParam<?> op,
                      final Param<?, ?>... parameters) {
    super.init(ugi, delegation, username, doAsUser, path, op, parameters);

    remoteAddr = JspHelper.getRemoteAddr(request);
  }

  @Override
  protected ClientProtocol getRpcClientProtocol() throws IOException {
    final Router router = getRouter();
    final RouterRpcServer routerRpcServer = router.getRpcServer();
    if (routerRpcServer == null) {
      throw new RetriableException("Router is in startup mode");
    }
    return routerRpcServer;
  }

  private void reset() {
    remoteAddr = null;
  }

  @Override
  protected String getRemoteAddr() {
    return remoteAddr;
  }

  @Override
  protected void queueExternalCall(ExternalCall call)
      throws IOException, InterruptedException {
    getRouter().getRpcServer().getServer().queueCall(call);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the WebHDFS request with backoff until the RPC server reports up (RetriableException is explicitly safe to retry)
2. Gate traffic on the Router being out of safe mode and its RPC server running (check dfsrouteradmin -safemode get / router JMX) before sending client requests
3. In HA setups point the client/LB at another already-running Router while one restarts

Example fix

// before
Response r = routerWebHdfs.get(url); // may fail during router startup

// after
for (int i = 0; i < MAX_RETRIES; i++) {
  try {
    Response r = routerWebHdfs.get(url);
    return r;
  } catch (RetriableException e) { // "Router is in startup mode"
    Thread.sleep(1000L << i);
  }
}

catch (RetriableException e) { // "Router is in startup mode"
  // schedule retry with backoff; do not treat as permanent failure
}

When it happens

Trigger: WebHDFS calls (any op via the Router's dfs.http.router address) issued within the seconds between the Router HTTP server binding and the RPC server starting; requests during Router restart; requests right after process start while the State Store connection is still being established.

Common situations: Scripts or monitoring probes that hit the Router the moment it listens; rolling restarts of Routers behind a load balancer; health checks keyed only on HTTP port availability.

Related errors

• {} is not supported
• Failed to find datanode, suggest to check cluster health. ex
• File {} not found.
• Offset={} out of the range [0, {}); {}, path={}
• Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d3c2f095034179b3. Report an issue: GitHub.