{"record":{"id":"d5f9bea1dfb91eef","repo":"jhy/jsoup","slug":"http-error-fetching-url","errorCode":null,"errorMessage":"HTTP error fetching URL","messagePattern":"HTTP error fetching URL","errorType":"http","errorClass":"HttpStatusException","httpStatus":null,"severity":"error","filePath":"src/main/java/org/jsoup/helper/HttpConnection.java","lineNumber":956,"sourceCode":"                            req.removeHeader(header);\n                    }\n\n                    String location = res.header(LOCATION);\n                    Validate.notNull(location);\n                    URL redir = StringUtil.resolve(req.url(), location);\n                    if (!sameOrigin(req.url(), redir)) {\n                        // remove sensitive headers; defense-in-depth against open redirects\n                        req.removeHeader(\"Authorization\");\n                        req.removeHeader(\"Cookie\");\n                        req.removeHeader(\"Cookie2\");\n                        req.cookies().clear();\n                    }\n                    req.url(redir);\n\n                    return execute(req, res);\n                }\n                if ((res.statusCode < 200 || res.statusCode >= 400) && !req.ignoreHttpErrors())\n                        throw new HttpStatusException(\"HTTP error fetching URL\", res.statusCode, req.url().toString());\n\n                // check that we can handle the returned content type; if not, abort before fetching it\n                String contentType = res.contentType();\n                boolean isText = contentType != null && contentType.regionMatches(true, 0, \"text/\", 0, 5);\n                boolean isXml = contentType != null && xmlContentTypeRxp.matcher(contentType).matches();\n\n                if (contentType != null\n                        && !req.ignoreContentType()\n                        && !isText\n                        && !isXml\n                        )\n                    throw new UnsupportedMimeTypeException(\"Unhandled content type. Must be a text or XML media type\",\n                            contentType, req.url().toString());\n\n                // switch to the XML parser if content type is xml and not parser not explicitly set\n                if (isXml) {\n                    if (!req.parserDefined) req.parser(Parser.xmlParser());\n                }","sourceCodeStart":938,"sourceCodeEnd":974,"githubUrl":"https://github.com/jhy/jsoup/blob/9851ac5d9c576c6888910b5a51a2362bbc978959/src/main/java/org/jsoup/helper/HttpConnection.java#L938-L974","documentation":"Jsoup throws HttpStatusException with message 'HTTP error fetching URL' when the response status is outside 200-399 and ignoreHttpErrors(false) (the default). It is jsoup's way of surfacing 4xx/5xx server responses instead of returning a document. The exception carries the status code and URL.","triggerScenarios":"Executing a request that returns 404, 500, 403, etc., without calling ignoreHttpErrors(true); scraping URLs that have moved or been removed; hitting endpoints requiring auth.","commonSituations":"Dead links in crawls; expired/invalid session cookies yielding 401/403; rate-limiting (429) from aggressive scraping; server errors during deploys.","solutions":["Check the URL and fix the request (headers, auth, cookies) to get a 2xx response","Catch org.jsoup.HttpStatusException and inspect getStatusCode() to handle statuses individually","Call ignoreHttpErrors(true) if you intend to parse error pages yourself","Add retry/backoff for transient 5xx/429 responses"],"exampleFix":"// before\nDocument doc = Jsoup.connect(url).get(); // throws on 404\n// after\ntry {\n    Document doc = Jsoup.connect(url).get();\n} catch (HttpStatusException e) {\n    if (e.getStatusCode() == 404) { /* handle missing page */ }\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { Document doc = Jsoup.connect(url).get(); } catch (HttpStatusException e) { switch (e.getStatusCode()) { case 404: /* skip */; case 429: /* backoff */; default: /* log */ } }","preventionTips":["Always catch HttpStatusException around fetches in crawlers","Implement backoff for 429/5xx","Check auth/cookies when seeing 401/403","Use ignoreHttpErrors(true) only when you deliberately parse error pages"],"tags":["java","jsoup","http","status-code"],"backgroundTag":"http-error-status","analyzedSha":"9851ac5d9c576c6888910b5a51a2362bbc978959","analyzedAt":"2026-09-08T15:22:04.931Z","contentChangedAt":"2026-09-08T15:22:04.931Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}