{"record":{"id":"5e1aa09b86f064fe","repo":"apache/hadoop","slug":"fetch-of-url-failed-with-status-code-connection-ge","errorCode":null,"errorMessage":"Fetch of url failed with status code connection.getResponseCode()\\nResponse message:\\nconnection.getResponseMessage()","messagePattern":"Fetch of url failed with status code connection\\.getResponseCode\\(\\)\\\\nResponse message:\\\\nconnection\\.getResponseMessage\\(\\)","errorType":"exception","errorClass":"HttpGetFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java","lineNumber":494,"sourceCode":"      this.url = url;\n    }\n\n    @Override\n    public InputStream getInputStream() throws IOException {\n      return SecurityUtil.doAsCurrentUser(\n          new PrivilegedExceptionAction<InputStream>() {\n            @Override\n            public InputStream run() throws IOException {\n              HttpURLConnection connection;\n              try {\n                connection = (HttpURLConnection)\n                    connectionFactory.openConnection(url, isSpnegoEnabled);\n              } catch (AuthenticationException e) {\n                throw new IOException(e);\n              }\n              \n              if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {\n                throw new HttpGetFailedException(\n                    \"Fetch of \" + url +\n                    \" failed with status code \" + connection.getResponseCode() +\n                    \"\\nResponse message:\\n\" + connection.getResponseMessage(),\n                    connection);\n              }\n        \n              String contentLength = connection.getHeaderField(CONTENT_LENGTH);\n              if (contentLength != null) {\n                advertisedSize = Long.parseLong(contentLength);\n                if (advertisedSize <= 0) {\n                  throw new IOException(\"Invalid \" + CONTENT_LENGTH + \" header: \" +\n                      contentLength);\n                }\n              } else {\n                throw new IOException(CONTENT_LENGTH + \" header is not provided \" +\n                                      \"by the server when trying to fetch \" + url);\n              }\n        ","sourceCodeStart":476,"sourceCodeEnd":512,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java#L476-L512","documentation":"HttpGetFailedException thrown by the URLLog fetcher inside EditLogFileInputStream: a remote edit log is opened over HTTP (e.g., the EditLogTailer on a standby/observer NameNode pulling segments from a peer's GetImageServlet), and connection.getResponseCode() != 200. The message carries the exact URL, status code, and the server's response message, which identifies the real cause.","triggerScenarios":"Opening EditLogFileInputStream on an http(s) URL to the journal endpoint and getting 404 (segment not yet available on that NN or already purged), 401/403 (SPNEGO/Kerberos failure), or 5xx (servlet error on the source NN). Reachable only from NN-internal tailing or tooling that fetches edits over HTTP.","commonSituations":"Standby tailer lagging until segments were purged; Kerberos keytab/krb5.conf/SPN (HTTP/host@REALM) misconfigured on the tailing side; dfs.namenode.http-address pointing at the wrong host; a firewall or load balancer in the path returning 502/503.","solutions":["Reproduce by hand with the URL from the log: kinit, then curl --negotiate -sv '<getedit URL>' -o /dev/null and read the status/message.","404: confirm the segment exists on the source NN (ls current/ or journal report); if simply not ready yet, the tailer retries on its interval -- check edit-lag metrics.","401/403: fix SPNEGO (keytab present, dfs.namenode.kerberos.principal/SPN HTTP/nn@REALM, krb5.conf, clock skew).","5xx: read the source NN's logs for the servlet failure; fix NN health or version mismatch on GetImageServlet parameters."],"exampleFix":"# before: tailer log shows Fetch of http://nn1:9870/getimage?getedit=1&txid=... failed with status code 404\nkinit -kt /etc/security/keytab/nn.keytool nn/nn1@REALM\ncurl --negotiate -s -o /dev/null -w '%{http_code}\\n' 'http://nn1:9870/getimage?getedit=1&txid=1234'\n# after: 200 -> segment now available, tailer recovers on next cycle; 403 -> fix SPNEGO","handlingStrategy":"retry","validationCode":"kinit -kt /etc/security/keytab/nn.keytab nn/nn1@REALM\ncurl --negotiate -s -o /dev/null -w '%{http_code}\\n' \"<the exact getedit URL from the NN log>\"\n# expect 200 before the tailer can succeed; 401/403 -> fix Kerberos, 404 -> segment absent","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e instanceof HttpGetFailedException h) {          // or match on message 'Fetch of '\n    int code = h.getStatusCode();                        // from the exception's connection\n    if (code == 404) { scheduleRetry(backoff); }         // segment not ready / purged\n    else if (code == 401 || code == 403) { alert(\"SPNEGO misconfigured\"); }\n    else { retryWithBackoff(); }\n  } else { throw e; }\n}","preventionTips":["Keep edit-segment retention longer than the standby/tailer's worst-case lag.","Monitor edit-tail lag metrics and alert before segments disappear.","Verify SPNEGO setup (keytab, SPN HTTP/host@REALM, krb5.conf, clock skew) after any Kerberos change.","Point tailers directly at NameNodes, not through proxies that rewrite status codes."],"tags":["hdfs","edit-log","http","kerberos","standby-nn"],"backgroundTag":"http-error-status-code","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}