{"record":{"id":"d3c2f095034179b3","repo":"apache/hadoop","slug":"router-is-in-startup-mode","errorCode":null,"errorMessage":"Router is in startup mode","messagePattern":"Router is in startup mode","errorType":"exception","errorClass":"RetriableException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java","lineNumber":155,"sourceCode":"  }\n\n  @Override\n  protected void init(final UserGroupInformation ugi,\n                      final DelegationParam delegation,\n                      final UserParam username, final DoAsParam doAsUser,\n                      final UriFsPathParam path, final HttpOpParam<?> op,\n                      final Param<?, ?>... parameters) {\n    super.init(ugi, delegation, username, doAsUser, path, op, parameters);\n\n    remoteAddr = JspHelper.getRemoteAddr(request);\n  }\n\n  @Override\n  protected ClientProtocol getRpcClientProtocol() throws IOException {\n    final Router router = getRouter();\n    final RouterRpcServer routerRpcServer = router.getRpcServer();\n    if (routerRpcServer == null) {\n      throw new RetriableException(\"Router is in startup mode\");\n    }\n    return routerRpcServer;\n  }\n\n  private void reset() {\n    remoteAddr = null;\n  }\n\n  @Override\n  protected String getRemoteAddr() {\n    return remoteAddr;\n  }\n\n  @Override\n  protected void queueExternalCall(ExternalCall call)\n      throws IOException, InterruptedException {\n    getRouter().getRpcServer().getServer().queueCall(call);\n  }","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java#L137-L173","documentation":"RouterWebHdfsMethods.getRpcClientProtocol throws RetriableException when a WebHDFS request reaches the Router before its RPC server has been created (router.getRpcServer() returns null). It signals a startup race, not a permanent failure: the HTTP endpoint (Router WebHDFS) is up while the core RPC service is still initializing. RetriableException tells well-behaved clients to retry the request later.","triggerScenarios":"WebHDFS calls (any op via the Router's dfs.http.router address) issued within the seconds between the Router HTTP server binding and the RPC server starting; requests during Router restart; requests right after process start while the State Store connection is still being established.","commonSituations":"Scripts or monitoring probes that hit the Router the moment it listens; rolling restarts of Routers behind a load balancer; health checks keyed only on HTTP port availability.","solutions":["Retry the WebHDFS request with backoff until the RPC server reports up (RetriableException is explicitly safe to retry)","Gate traffic on the Router being out of safe mode and its RPC server running (check dfsrouteradmin -safemode get / router JMX) before sending client requests","In HA setups point the client/LB at another already-running Router while one restarts"],"exampleFix":"// before\nResponse r = routerWebHdfs.get(url); // may fail during router startup\n\n// after\nfor (int i = 0; i < MAX_RETRIES; i++) {\n  try {\n    Response r = routerWebHdfs.get(url);\n    return r;\n  } catch (RetriableException e) { // \"Router is in startup mode\"\n    Thread.sleep(1000L << i);\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (RetriableException e) { // \"Router is in startup mode\"\n  // schedule retry with backoff; do not treat as permanent failure\n}","preventionTips":["Treat HTTP port liveness of the Router as insufficient readiness; wait for RPC server up / safe mode off","Use Router HA endpoints so a starting Router is bypassed","Probes and cron jobs should tolerate a short startup window with retries"],"tags":["hadoop","hdfs","rbf","webhdfs","startup","retriable"],"backgroundTag":"service-unavailable-retry","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}