jhy/jsoup · error · HttpStatusException

HTTP error fetching URL

Error message

HTTP error fetching URL

What it means

Jsoup throws HttpStatusException with message 'HTTP error fetching URL' when the response status is outside 200-399 and ignoreHttpErrors(false) (the default). It is jsoup's way of surfacing 4xx/5xx server responses instead of returning a document. The exception carries the status code and URL.

Solutions

1. Check the URL and fix the request (headers, auth, cookies) to get a 2xx response
2. Catch org.jsoup.HttpStatusException and inspect getStatusCode() to handle statuses individually
3. Call ignoreHttpErrors(true) if you intend to parse error pages yourself
4. Add retry/backoff for transient 5xx/429 responses

Example fix

// before
Document doc = Jsoup.connect(url).get(); // throws on 404
// after
try {
    Document doc = Jsoup.connect(url).get();
} catch (HttpStatusException e) {
    if (e.getStatusCode() == 404) { /* handle missing page */ }
}

try { Document doc = Jsoup.connect(url).get(); } catch (HttpStatusException e) { switch (e.getStatusCode()) { case 404: /* skip */; case 429: /* backoff */; default: /* log */ } }

When it happens

Trigger: Executing a request that returns 404, 500, 403, etc., without calling ignoreHttpErrors(true); scraping URLs that have moved or been removed; hitting endpoints requiring auth.

Common situations: Dead links in crawls; expired/invalid session cookies yielding 401/403; rate-limiting (429) from aggressive scraping; server errors during deploys.

Related errors

• You must execute the request before getting a response.
• URL not set. Make sure to call #url(...) before executing…
• Cannot follow redirect with a streamed request body…
• Unhandled content type. Must be a text or XML media type
• Too many redirects occurred trying to load URL

AI-assisted analysis of jhy/jsoup@9851ac5d9c (2026-09-08). Data as JSON: /api/errors/d5f9bea1dfb91eef. Report an issue: GitHub.

Appendix: source

Thrown at src/main/java/org/jsoup/helper/HttpConnection.java:956

                            req.removeHeader(header);
                    }

                    String location = res.header(LOCATION);
                    Validate.notNull(location);
                    URL redir = StringUtil.resolve(req.url(), location);
                    if (!sameOrigin(req.url(), redir)) {
                        // remove sensitive headers; defense-in-depth against open redirects
                        req.removeHeader("Authorization");
                        req.removeHeader("Cookie");
                        req.removeHeader("Cookie2");
                        req.cookies().clear();
                    }
                    req.url(redir);

                    return execute(req, res);
                }
                if ((res.statusCode < 200 || res.statusCode >= 400) && !req.ignoreHttpErrors())
                        throw new HttpStatusException("HTTP error fetching URL", res.statusCode, req.url().toString());

                // check that we can handle the returned content type; if not, abort before fetching it
                String contentType = res.contentType();
                boolean isText = contentType != null && contentType.regionMatches(true, 0, "text/", 0, 5);
                boolean isXml = contentType != null && xmlContentTypeRxp.matcher(contentType).matches();

                if (contentType != null
                        && !req.ignoreContentType()
                        && !isText
                        && !isXml
                        )
                    throw new UnsupportedMimeTypeException("Unhandled content type. Must be a text or XML media type",
                            contentType, req.url().toString());

                // switch to the XML parser if content type is xml and not parser not explicitly set
                if (isXml) {
                    if (!req.parserDefined) req.parser(Parser.xmlParser());
                }

View on GitHub (pinned to 9851ac5d9c)