apify/crawlee · error · Error

${status} - ${message}

Error message

${status} - ${message}

What it means

When a response has an error status code (>= 500 or in blockedStatusCodes / additionalHttpErrorStatusCodes handling path) and Content-Type application/json, parseResponse JSON-parses the body and throws `${status} - ${message}` using the JSON's `message` field (or util.inspect of the whole payload if absent). This surfaces the server's own error description to the retry/error machinery.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:737

        const { status } = response;
        const { type, charset } = parseContentTypeFromResponse(response);
        const { response: reencodedResponse, encoding } = this.encodeResponse(request, response, charset);
        const contentType = { type, encoding };

        if (status >= 400 && status <= 599) {
            this.statistics.registerStatusCode(status);
        }

        if (this.isErrorStatusCode(status)) {
            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)

            // Errors are often sent as JSON, so attempt to parse them,
            // despite Accept header being set to text/html.
            if (type === APPLICATION_JSON_MIME_TYPE) {
                const errorResponse = JSON.parse(body);
                let { message } = errorResponse;
                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });
                throw new Error(`${status} - ${message}`);
            }

            if (this.additionalHttpErrorStatusCodes.has(status)) {
                throw new Error(`${status} - Error status code was set by user.`);
            }

            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.
            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);
        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {
            if (!charset && !this.#forceResponseEncoding) {
                const rawBytes = Buffer.from(await response.arrayBuffer());
                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);
                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';
                const body = iconv.encodingExists(charsetToUse)
                    ? iconv.decode(rawBytes, charsetToUse)
                    : rawBytes.toString('utf8');
                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };
            }

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Inspect the embedded message to see what the server actually complained about and fix that root cause (payload, auth, rate).
2. Configure the crawler's retry settings so transient 5xx responses are retried with backoff.
3. If the status is actually acceptable for your use case, add it via `ignoreHttpErrors` / adjust `additionalHttpErrorStatusCodes` / `blockedStatusCodes`.
4. Implement `postNavigationHooks` or a custom requestFunction to handle known JSON error contracts gracefully.

Example fix

// before
new HttpCrawler({ additionalHttpErrorStatusCodes: [429] });

// after: accept 429 responses instead of throwing
new HttpCrawler({ ignoreHttpErrors: false, additionalHttpErrorStatusCodes: [], ignoreSslErrors: false,
  // handle 429 by retrying later
  failedRequestHandler: async ({ request, error }) => log.error(`${request.url}: ${error.message}`),
});

const res = await got(url, { throwHttpErrors: false, responseType: 'json' });
if (res.statusCode >= 500 && res.body?.message) {
  log.warning(`Server reports: ${res.body.message}`);
}

function isStatusJsonError(err: unknown, status: number): boolean {
  return err instanceof Error && err.message.startsWith(`${status} - `);
}

try {
  await crawler.run(requests);
} catch (err) {
  const m = /^\d{3} - /.exec(err instanceof Error ? err.message : '');
  if (m) {
    log.warning(`HTTP error response, will rely on crawler retry: ${err.message}`);
  } else {
    throw err;
  }
}

When it happens

Trigger: The target responds with a non-2xx status (e.g. 500, 502, 503) and JSON body like {"message":"upstream timeout"}; the crawler's statusCode validation converts it into this thrown Error.

Common situations: API endpoints returning JSON errors for rate limiting or server faults; gateways (Cloudflare/nginx) emitting JSON error payloads; temporary upstream outages while crawling APIs.

Related errors

• ${status} - Internal Server Error: ${body.slice(0, 100)}
• ${status} - Error status code was set by user.
• Could not parse cookie header string: ${cookieHeaderString}
• Request with GET method cannot have a payload.
• Data item${s}is not an object. You can push only objects int

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/a5501eb2eee38d5a. Report an issue: GitHub.