{"record":{"id":"cea57c1ad4459ca3","repo":"apache/hadoop","slug":"namenode-initialization-not-yet-complete-fsimage","errorCode":null,"errorMessage":"NameNode initialization not yet complete. FSImage has not been set in the NameNode.","messagePattern":"NameNode initialization not yet complete\\. FSImage has not been set in the NameNode\\.","errorType":"http","errorClass":"IOException","httpStatus":403,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java","lineNumber":124,"sourceCode":"   * due to image upload delay, or minor machine clock skew) can cause ANN to\n   * reject a fsImage too aggressively.\n   */\n  private static double recentImageCheckTimePrecision = 0.75;\n\n  @VisibleForTesting\n  static void setRecentImageCheckTimePrecision(double ratio) {\n    recentImageCheckTimePrecision = ratio;\n  }\n\n  private FSImage getAndValidateFSImage(ServletContext context,\n      final HttpServletResponse response)\n      throws IOException {\n    final FSImage nnImage = NameNodeHttpServer.getFsImageFromContext(context);\n    if (nnImage == null) {\n      String errorMsg = \"NameNode initialization not yet complete. \"\n          + \"FSImage has not been set in the NameNode.\";\n      sendError(response, HttpServletResponse.SC_FORBIDDEN, errorMsg);\n      throw new IOException(errorMsg);\n    }\n    return nnImage;\n  }\n\n  @Override\n  public void doGet(final HttpServletRequest request,\n      final HttpServletResponse response) throws ServletException, IOException {\n    try {\n      final ServletContext context = getServletContext();\n      final FSImage nnImage = getAndValidateFSImage(context, response);\n      final GetImageParams parsedParams = new GetImageParams(request, response);\n      final Configuration conf = (Configuration) context\n          .getAttribute(JspHelper.CURRENT_CONF);\n      final NameNodeMetrics metrics = NameNode.getNameNodeMetrics();\n\n      validateRequest(context, conf, request, response, nnImage,\n          parsedParams.getStorageInfoString());\n","sourceCodeStart":106,"sourceCodeEnd":142,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java#L106-L142","documentation":"ImageServlet.getAndValidateFSImage returns HTTP 403 and throws IOException(\"NameNode initialization not yet complete. FSImage has not been set in the NameNode.\") when an image-transfer HTTP request arrives after the NameNode's HTTP port is open but before the FSImage object has been published to the servlet context. It is a startup-window race: the endpoint exists but is not yet serviceable.","triggerScenarios":"SecondaryNameNode or a bootstrapping Standby immediately requests /getimage right after the NN process restarts; monitoring/backup scripts polling the image-transfer endpoint during a long fsimage load; NN restart storms where checkpointers retry aggressively.","commonSituations":"Restart automation without readiness gating; very large fsimages extending the load window while SNN timers fire; health checks that hit transfer servlets instead of status endpoints.","solutions":["Retry the request after the NameNode finishes startup — poll /jmx (State, Safemode) or HA state until it is ready","Gate automated checkpoints/bootstrap on NN readiness rather than fixed timers","If the error persists well after startup, check NN logs: the image may be failing to load entirely"],"exampleFix":"# before\ncurl -f http://nn:9870/getimage?getimage=1   # during NN startup -> 403\n\n# after: gate on readiness, then transfer\nuntil curl -sf http://nn:9870/jmx | grep -q '\"State\" : \"active\"'; do sleep 5; done\ncurl -f http://nn:9870/getimage?getimage=1","handlingStrategy":"retry","validationCode":"// Gate image-transfer scripts on NN readiness before the first request\n# bash\nuntil curl -sf http://nn:9870/jmx \\\n  | grep -q '\"State\" : \"\\(active\\|standby\\)\"'; do\n  sleep 5\ndone\ncurl -f 'http://nn:9870/getimage?getimage=1&latest=1'","typeGuard":null,"tryCatchPattern":"int attempts = 12;\nfor (int i = 0; i < attempts; i++) {\n  try {\n    return fetchImage(url);\n  } catch (IOException e) {\n    if (!e.getMessage().contains(\"initialization not yet complete\")\n        || i == attempts - 1) { throw e; }\n    Thread.sleep(10_000L);   // NN still loading: back off and retry\n  }\n}","preventionTips":["Sequence SNN checkpoints and bootstrap-standby after NN readiness checks, not fixed delays","Monitor NameNode startup progress (/jmx StartupProgress) during long image loads","Treat 403 'initialization not yet complete' as a retryable signal in tooling, distinct from auth 403s"],"tags":["hdfs","namenode","startup","checkpoint","http-403","image-transfer"],"backgroundTag":"service-not-ready","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}