{"record":{"id":"043e188675c91db2","repo":"apify/crawlee","slug":"status-error-status-code-was-set-by-user","errorCode":null,"errorMessage":"${status} - Error status code was set by user.","messagePattern":"(.+?) - Error status code was set by user\\.","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/browser-crawler/src/internals/browser-crawler.ts","lineNumber":853,"sourceCode":"\n        if (typeof response === 'object' && typeof response.status === 'function') {\n            const status: number = response.status();\n\n            this.statistics.registerStatusCode(status);\n\n            // Ahead of the error-status throw below: a 429 the user opted into treating as an error is still a\n            // rate limit the domain should back off from.\n            if (status === 429) {\n                // Both drivers lower-case header names and join duplicates, so a plain lookup is enough.\n                const retryAfter = response.headers?.()['retry-after'];\n                if (this.recordDomainRateLimit(crawlingContext.request.url, retryAfter)) {\n                    throw new RequestThrottledError(`${crawlingContext.request.url} responded with 429.`);\n                }\n            }\n\n            if (this.isErrorStatusCode(status)) {\n                if (this.additionalHttpErrorStatusCodes.has(status)) {\n                    throw new Error(`${status} - Error status code was set by user.`);\n                }\n\n                throw new Error(`${status} - Internal Server Error`);\n            }\n        }\n\n        if (this.sessionPool && response && session) {\n            if (typeof response === 'object' && typeof response.status === 'function') {\n                this.throwOnBlockedRequest(response.status());\n            } else {\n                this.log.debug('Got a malformed Browser response.', { request, response });\n            }\n        }\n\n        request.loadedUrl = await page.url();\n    }\n\n    /**","sourceCodeStart":835,"sourceCodeEnd":871,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/browser-crawler/src/internals/browser-crawler.ts#L835-L871","documentation":"`processResponse()` treats any status that `isErrorStatusCode` returns true for as a failed navigation. If the specific status is in `additionalHttpErrorStatusCodes` — statuses the user added beyond the defaults — the thrown error message explicitly says the user configured it, distinguishing it from the generic internal-server-error case. The request fails and is retried up to `maxRequestRetries`.","triggerScenarios":"The site responds with a status code that was explicitly added via crawler option `additionalHttpErrorStatusCodes` (e.g. `[404, 203]`), and `isErrorStatusCode` flags it — for example a status >= 500 or a user-configured error status matching this branch.","commonSituations":"Developers add a status like 404 or 403 to force retries/retirement but forget that their own site legitimately returns it (soft-404 pages, expiring listings), causing all such requests to fail; migrating to a site that returns nonstandard success codes (e.g. 203) that were added as error codes earlier.","solutions":["Remove the offending status from `additionalHttpErrorStatusCodes` if it should not be treated as an error","Keep the status configured but handle it in your `requestHandler` — the error is thrown before the handler runs, so instead filter such URLs out in `preNavigationHooks` or skip them at enqueue time","Use `request.noRetry`/error handling in `failedRequestHandler` to accept these responses without retry storms","Check the actual status code in the message against what your target site legitimately returns"],"exampleFix":"// before\nconst crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [404] }); // site legitimately returns 404 for empty results\n// after\nconst crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [500] }); // only treat 5xx as hard errors","handlingStrategy":"validation","validationCode":"// Before the crawl, confirm every configured error status is one you truly want to fail:\nconst crawler = new PlaywrightCrawler({ additionalHttpErrorStatusCodes: [404] });\nconst defaults = [500, 502, 503, 504];\nconst allErrorStatuses = [...defaults, ...crawler.additionalHttpErrorStatusCodes];\nconsole.assert(!allErrorStatuses.includes(200) && allErrorStatuses.length < 20, 'Review additionalHttpErrorStatusCodes');","typeGuard":null,"tryCatchPattern":"crawler.failedRequestHandler = async ({ request, error }) => {\n    if (/was set by user$/.test(error.message)) {\n        const status = Number(error.message.split(' ')[0]);\n        log.warning(`User-configured error status ${status} on ${request.url}`);\n    }\n};","preventionTips":["Only add statuses to additionalHttpErrorStatusCodes that the target should never legitimately return","Re-audit the list whenever the target site changes its API responses","Prefer filtering unwanted URLs at enqueue time over declaring them error statuses"],"tags":["http-status","error-status-code","retry","configuration"],"backgroundTag":"http-error-status-code","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}