What causes "${status} - Error status code was set by user." in apify/crawlee?

Remove the offending status from `additionalHttpErrorStatusCodes` if it should not be treated as an error

apify/crawlee · error · Error

${status} - Error status code was set by user.

Error message

${status} - Error status code was set by user.

What it means

`processResponse()` treats any status that `isErrorStatusCode` returns true for as a failed navigation. If the specific status is in `additionalHttpErrorStatusCodes` — statuses the user added beyond the defaults — the thrown error message explicitly says the user configured it, distinguishing it from the generic internal-server-error case. The request fails and is retried up to `maxRequestRetries`.

Source

Thrown at packages/browser-crawler/src/internals/browser-crawler.ts:853

        if (typeof response === 'object' && typeof response.status === 'function') {
            const status: number = response.status();

            this.statistics.registerStatusCode(status);

            // Ahead of the error-status throw below: a 429 the user opted into treating as an error is still a
            // rate limit the domain should back off from.
            if (status === 429) {
                // Both drivers lower-case header names and join duplicates, so a plain lookup is enough.
                const retryAfter = response.headers?.()['retry-after'];
                if (this.recordDomainRateLimit(crawlingContext.request.url, retryAfter)) {
                    throw new RequestThrottledError(`${crawlingContext.request.url} responded with 429.`);
                }
            }

            if (this.isErrorStatusCode(status)) {
                if (this.additionalHttpErrorStatusCodes.has(status)) {
                    throw new Error(`${status} - Error status code was set by user.`);
                }

                throw new Error(`${status} - Internal Server Error`);
            }
        }

        if (this.sessionPool && response && session) {
            if (typeof response === 'object' && typeof response.status === 'function') {
                this.throwOnBlockedRequest(response.status());
            } else {
                this.log.debug('Got a malformed Browser response.', { request, response });
            }
        }

        request.loadedUrl = await page.url();
    }

    /**

View on GitHub (pinned to dbe57fb09c)

Solutions

Remove the offending status from `additionalHttpErrorStatusCodes` if it should not be treated as an error
Keep the status configured but handle it in your `requestHandler` — the error is thrown before the handler runs, so instead filter such URLs out in `preNavigationHooks` or skip them at enqueue time
Use `request.noRetry`/error handling in `failedRequestHandler` to accept these responses without retry storms
Check the actual status code in the message against what your target site legitimately returns

Example fix

// before
const crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [404] }); // site legitimately returns 404 for empty results
// after
const crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [500] }); // only treat 5xx as hard errors

Defensive patterns

Strategy: validation

Validate before calling

// Before the crawl, confirm every configured error status is one you truly want to fail:
const crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [404] });
const defaults = [500, 502, 503, 504];
const allErrorStatuses = [...defaults, ...crawler.additionalHttpErrorStatusCodes];
console.assert(!allErrorStatuses.includes(200) && allErrorStatuses.length < 20, 'Review additionalHttpErrorStatusCodes');

Try / catch

crawler.failedRequestHandler = async ({ request, error }) => {
    if (/was set by user$/.test(error.message)) {
        const status = Number(error.message.split(' ')[0]);
        log.warning(`User-configured error status ${status} on ${request.url}`);
    }
};

Prevention

Only add statuses to additionalHttpErrorStatusCodes that the target should never legitimately return
Re-audit the list whenever the target site changes its API responses
Prefer filtering unwanted URLs at enqueue time over declaring them error statuses

When it happens

Trigger: The site responds with a status code that was explicitly added via crawler option `additionalHttpErrorStatusCodes` (e.g. `[404, 203]`), and `isErrorStatusCode` flags it — for example a status >= 500 or a user-configured error status matching this branch.

Common situations: Developers add a status like 404 or 403 to force retries/retirement but forget that their own site legitimately returns it (soft-404 pages, expiring listings), causing all such requests to fail; migrating to a site that returns nonstandard success codes (e.g. 203) that were added as error codes earlier.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/043e188675c91db2. Report an issue: GitHub.