apify/crawlee · error · Error

${status} - Error status code was set by user.

Error message

${status} - Error status code was set by user.

What it means

If the error status code was explicitly added by the user via `additionalHttpErrorStatusCodes`, parseResponse throws `${status} - Error status code was set by user.` because the response body is not JSON and no server message is available. This makes user-designated status codes fail the request even if they would otherwise be treated as success.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:741

        if (status >= 400 && status <= 599) {
            this.statistics.registerStatusCode(status);
        }

        if (this.isErrorStatusCode(status)) {
            const body = await reencodedResponse.text(); // TODO - this always uses UTF-8 (see https://developer.mozilla.org/en-US/docs/Web/API/Request/text)

            // Errors are often sent as JSON, so attempt to parse them,
            // despite Accept header being set to text/html.
            if (type === APPLICATION_JSON_MIME_TYPE) {
                const errorResponse = JSON.parse(body);
                let { message } = errorResponse;
                if (!message) message = util.inspect(errorResponse, { depth: 1, maxArrayLength: 10 });
                throw new Error(`${status} - ${message}`);
            }

            if (this.additionalHttpErrorStatusCodes.has(status)) {
                throw new Error(`${status} - Error status code was set by user.`);
            }

            // It's not a JSON, so it's probably some text. Get the first 100 chars of it.
            throw new Error(`${status} - Internal Server Error: ${body.slice(0, 100)}`);
        } else if (HTML_AND_XML_MIME_TYPES.includes(type)) {
            if (!charset && !this.#forceResponseEncoding) {
                const rawBytes = Buffer.from(await response.arrayBuffer());
                const metaCharset = extractCharsetFromHtmlBytes(rawBytes);
                const charsetToUse = metaCharset ?? this.#suggestResponseEncoding ?? 'utf-8';
                const body = iconv.encodingExists(charsetToUse)
                    ? iconv.decode(rawBytes, charsetToUse)
                    : rawBytes.toString('utf8');
                return { response, contentType: { type, encoding: 'utf-8' as BufferEncoding }, body };
            }
            return { response, contentType, body: await reencodedResponse.text() };
        } else {
            const body = Buffer.from(await reencodedResponse.bytes());
            return {

View on GitHub (pinned to dbe57fb09c)

Solutions

  1. Confirm the added status code is really one you want treated as an error; remove it from additionalHttpErrorStatusCodes if not.
  2. Keep the code but handle it: check `error.message` in failedRequestHandler and skip logging as failure when expected.
  3. Use `ignoreHttpErrors` for statuses you want passed through without retry instead of adding them as errors.
  4. If the response is JSON, the first parse branch will produce a more descriptive message — no action needed.

Example fix

// before
new HttpCrawler({ additionalHttpErrorStatusCodes: [403] });

// after: treat 403 as blocked (session retry) instead of generic user error
new HttpCrawler({ retryOnBlocked: true, blockedStatusCodes: [403], useSessionPool: true });
Defensive patterns

Strategy: validation

Validate before calling

const crawler = new HttpCrawler({ additionalHttpErrorStatusCodes: [404] });
// audit config: only add statuses you truly want to fail requests
console.log('Treating as errors:', [404]);

Try / catch

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof Error && err.message.includes('Error status code was set by user.')) {
    const status = err.message.split(' - ')[0];
    log.info(`Expected user-flagged status ${status}; ignoring`);
    return;
  }
  throw err;
}

Prevention

When it happens

Trigger: Calling `new HttpCrawler({ additionalHttpErrorStatusCodes: [404, 410, ...] })` (or extendHttpCrawlerOptions equivalents) and receiving one of those statuses with a non-JSON body.

Common situations: Users marking soft-404 or bot-wall statuses as errors so they get retried; misremembering that additionalHttpErrorStatusCodes supplements (not replaces) the default error statuses; adding 2xx/3xx codes intentionally to force retries.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/cf37b7efdd746ba2. Report an issue: GitHub.